Sep 10 20:58:36.200922 ip-10-0-130-192 systemd[1]: Starting Kubernetes Kubelet... Sep 10 20:58:36.595097 ip-10-0-130-192 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:58:36.595097 ip-10-0-130-192 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 20:58:36.595097 ip-10-0-130-192 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:58:36.595097 ip-10-0-130-192 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 20:58:36.595097 ip-10-0-130-192 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:58:36.597990 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.597904 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 20:58:36.602210 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602195 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.602210 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602211 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602215 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602218 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602221 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602224 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602229 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602232 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602235 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602238 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602241 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602243 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602246 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602248 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602251 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602253 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602256 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602259 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602263 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602267 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602271 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.602272 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602275 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602278 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602281 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602284 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602287 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602290 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602294 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602297 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602300 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602302 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602305 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602308 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602310 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602313 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602315 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602318 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602320 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602323 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602326 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602328 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.602807 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602331 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602333 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602336 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602338 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602341 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602343 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602346 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602348 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602351 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602353 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602356 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602358 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602360 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602364 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602368 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602371 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602374 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602377 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602379 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.603298 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602382 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602386 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602389 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602392 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602395 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602398 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602401 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602403 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602406 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602408 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602411 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602413 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602416 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602418 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602421 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602424 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602427 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602430 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602432 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602435 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.603776 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602437 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602440 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602442 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602445 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602448 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602451 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602453 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602457 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602460 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602463 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602465 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602468 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602471 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602473 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602476 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602479 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602481 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602484 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.602487 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.604253 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603309 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603316 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603326 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603329 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603332 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603335 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603345 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603349 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603351 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603354 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603357 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603360 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603362 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603365 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603367 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603370 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603373 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603375 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603378 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603381 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.604728 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603384 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603387 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603389 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603392 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603394 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603397 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603400 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603402 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603405 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603407 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603410 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603413 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603415 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603418 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603420 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603423 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603425 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603428 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603430 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.605260 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603433 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603436 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603439 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603442 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603444 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603447 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603449 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603452 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603454 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603457 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603460 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603462 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603465 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603469 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603472 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603474 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603477 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603479 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603482 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603485 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.605742 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603487 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603490 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603492 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603495 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603499 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603502 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603506 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603509 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603511 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603515 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603519 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603522 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603524 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603527 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603530 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603532 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603535 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603539 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603543 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.606231 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603546 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603549 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603551 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603554 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603557 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603561 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603564 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603566 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603569 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603571 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603574 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603576 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603579 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603581 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603584 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603586 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603589 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603591 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603594 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603596 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.606710 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.603599 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604008 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604013 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604016 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604018 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604021 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604032 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604036 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604040 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604043 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604046 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604049 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604051 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604054 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604056 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604059 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604061 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604065 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604068 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.607197 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604071 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604073 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604076 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604079 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604081 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604084 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604086 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604089 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604091 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604094 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604096 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604099 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604101 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604104 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604107 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604109 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604111 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604114 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604117 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604119 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.607683 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604127 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604130 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604135 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604137 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604140 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604143 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604145 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604148 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604150 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604153 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604156 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604158 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604161 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604163 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604166 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604168 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604171 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604173 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604176 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604178 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.608180 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604181 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604183 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604187 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604191 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604193 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604196 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604198 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604201 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604203 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604206 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604209 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604211 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604213 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604221 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604225 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604228 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604230 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604233 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604235 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.608770 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604237 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604240 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604242 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604245 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604248 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604251 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604254 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604256 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604267 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604272 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604275 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604278 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604281 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604284 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604287 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604290 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604293 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604296 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604298 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604302 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.609250 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604304 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.604307 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604374 2585 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604407 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604413 2585 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604417 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604421 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604433 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604438 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604442 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604445 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604448 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604451 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604455 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604458 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604460 2585 flags.go:64] FLAG: --cgroup-root="" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604465 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604468 2585 flags.go:64] FLAG: --client-ca-file="" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604471 2585 flags.go:64] FLAG: --cloud-provider="external" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604474 2585 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604481 2585 flags.go:64] FLAG: --cluster-domain="" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604484 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604487 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604490 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 20:58:36.609748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604493 2585 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604497 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604500 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604503 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604507 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604510 2585 flags.go:64] FLAG: --contention-profiling="false" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604513 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604515 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604519 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604522 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604526 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604529 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604531 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604534 2585 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604537 2585 flags.go:64] FLAG: --enable-server="true" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604540 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604552 2585 flags.go:64] FLAG: --event-burst="100" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604555 2585 flags.go:64] FLAG: --event-qps="50" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604558 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604562 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604565 2585 flags.go:64] FLAG: --eviction-hard="" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604569 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604572 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604575 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604579 2585 flags.go:64] FLAG: --eviction-soft="" Sep 10 20:58:36.610332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604582 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604585 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604588 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604591 2585 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604594 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604597 2585 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604600 2585 flags.go:64] FLAG: --feature-gates="" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604604 2585 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604619 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604623 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604626 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604629 2585 flags.go:64] FLAG: --healthz-port="10248" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604632 2585 flags.go:64] FLAG: --help="false" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604635 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604638 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604641 2585 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604644 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604647 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604650 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604653 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604656 2585 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604659 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604662 2585 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 20:58:36.610983 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604665 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604674 2585 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604677 2585 flags.go:64] FLAG: --kube-reserved="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604680 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604683 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604686 2585 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604689 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604691 2585 flags.go:64] FLAG: --lock-file="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604695 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604698 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604701 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604707 2585 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604710 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604713 2585 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604716 2585 flags.go:64] FLAG: --logging-format="text" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604718 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604722 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604725 2585 flags.go:64] FLAG: --manifest-url="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604727 2585 flags.go:64] FLAG: --manifest-url-header="" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604732 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604735 2585 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604739 2585 flags.go:64] FLAG: --max-pods="110" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604742 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604745 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604748 2585 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 20:58:36.611574 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604751 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604754 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604757 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604760 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604773 2585 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604776 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604778 2585 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604781 2585 flags.go:64] FLAG: --pod-cidr="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604784 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604795 2585 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604797 2585 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604801 2585 flags.go:64] FLAG: --pods-per-core="0" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604803 2585 flags.go:64] FLAG: --port="10250" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604806 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604809 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-045945497a8f99d73" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604812 2585 flags.go:64] FLAG: --qos-reserved="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604816 2585 flags.go:64] FLAG: --read-only-port="10255" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604819 2585 flags.go:64] FLAG: --register-node="true" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604822 2585 flags.go:64] FLAG: --register-with-taints="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604825 2585 flags.go:64] FLAG: --registry-burst="10" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604828 2585 flags.go:64] FLAG: --registry-qps="5" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604831 2585 flags.go:64] FLAG: --reserved-cpus="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604833 2585 flags.go:64] FLAG: --reserved-memory="" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604837 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604841 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 20:58:36.612203 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604844 2585 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604853 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604856 2585 flags.go:64] FLAG: --runonce="false" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604859 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604861 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604865 2585 flags.go:64] FLAG: --seccomp-default="false" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604867 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604870 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604873 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604876 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604879 2585 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604882 2585 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604885 2585 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604888 2585 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604891 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604893 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604896 2585 flags.go:64] FLAG: --system-cgroups="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604904 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604910 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604913 2585 flags.go:64] FLAG: --tls-cert-file="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604915 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604924 2585 flags.go:64] FLAG: --tls-min-version="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604926 2585 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604929 2585 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604932 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 20:58:36.612828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604935 2585 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604938 2585 flags.go:64] FLAG: --v="2" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604942 2585 flags.go:64] FLAG: --version="false" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604946 2585 flags.go:64] FLAG: --vmodule="" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604949 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.604953 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605108 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605112 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605116 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605118 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605121 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605124 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605128 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605131 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605134 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605137 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605139 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605142 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605145 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605148 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605151 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.613485 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605153 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605156 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605158 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605161 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605170 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605173 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605175 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605178 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605181 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605187 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605189 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605192 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605194 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605197 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605201 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605204 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605207 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605212 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605215 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605217 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.614244 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605220 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605223 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605226 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605228 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605231 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605233 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605236 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605238 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605241 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605243 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605246 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605248 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605251 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605253 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605256 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605258 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605261 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605269 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605272 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605275 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.614923 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605277 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605280 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605282 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605285 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605288 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605290 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605293 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605295 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605298 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605302 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605304 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605307 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605309 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605312 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605314 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605317 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605319 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605321 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605324 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605327 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.615415 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605329 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605332 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605334 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605337 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605339 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605342 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605344 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605347 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605350 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605353 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605361 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605364 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605367 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605369 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605372 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605376 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605379 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605382 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605385 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.615933 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605388 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605391 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605396 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605398 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.605401 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.606157 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.615881 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.615898 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.615917 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.615923 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.615984 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.615989 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.615993 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.615997 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.616416 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616000 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616004 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616007 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616010 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616013 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616016 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616019 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616022 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616025 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616027 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616030 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616033 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616035 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616038 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616041 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616044 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616047 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616050 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616053 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616056 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.616790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616058 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616061 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616064 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616066 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616069 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616072 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616075 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616078 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616080 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616083 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616086 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616088 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616091 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616093 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616096 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616099 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616101 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616105 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616109 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616112 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.617268 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616115 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616117 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616131 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616134 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616137 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616140 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616142 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616145 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616148 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616151 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616154 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616156 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616159 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616161 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616164 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616166 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616169 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616172 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616175 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.617766 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616177 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616180 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616182 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616185 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616188 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616190 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616193 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616195 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616198 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616200 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616203 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616205 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616208 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616211 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616213 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616216 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616224 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616227 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616231 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.618256 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616235 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616238 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616241 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616244 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616247 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616249 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616252 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616255 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616257 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616269 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616272 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616275 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616277 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616280 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616283 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616287 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.618736 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616290 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.616295 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616409 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616414 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616417 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616421 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616424 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616427 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616430 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616434 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616436 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616439 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616442 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616445 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616456 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:58:36.619130 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616458 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616461 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616464 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616466 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616469 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616472 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616474 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616477 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616480 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616482 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616485 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616488 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616491 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616493 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616496 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616499 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616501 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616504 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616507 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616509 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:58:36.619506 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616513 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616518 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616520 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616523 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616526 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616529 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616531 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616534 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616536 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616539 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616542 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616545 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616548 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616551 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616554 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616557 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616560 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616562 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616565 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616568 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:58:36.620087 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616571 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616573 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616576 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616579 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616582 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616584 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616587 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616589 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616592 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616595 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616597 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616600 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616602 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616605 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616622 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616626 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616628 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616631 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616633 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616636 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:58:36.620572 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616639 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616641 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616644 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616647 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616650 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616653 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616656 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616659 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616662 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616664 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616667 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616670 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616672 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616675 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616679 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616683 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616687 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616690 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:58:36.621058 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616693 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616696 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616699 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616702 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616705 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616708 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616711 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:36.616713 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.616719 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.616828 2585 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 20:58:36.621501 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.621245 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 20:58:36.622278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.622265 2585 server.go:1019] "Starting client certificate rotation" Sep 10 20:58:36.622388 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.622371 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 20:58:36.622419 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.622413 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 20:58:36.625310 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.625296 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 20:58:36.626446 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.626428 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 20:58:36.638753 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.638730 2585 log.go:25] "Validated CRI v1 runtime API" Sep 10 20:58:36.642435 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.642420 2585 log.go:25] "Validated CRI v1 image API" Sep 10 20:58:36.643321 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.643299 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 20:58:36.650246 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.650222 2585 fs.go:135] Filesystem UUIDs: map[63842543-2274-4a98-8043-c9ae260a218e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a659ffad-2809-469e-9a0e-a8fff3405bbb:/dev/nvme0n1p4] Sep 10 20:58:36.650323 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.650245 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 20:58:36.657537 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.657414 2585 manager.go:216] Machine: {Timestamp:2026-09-10 20:58:36.655745783 +0000 UTC m=+0.364040186 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100090 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a4f202042787a0cb433cca9398cd4 SystemUUID:ec2a4f20-2042-787a-0cb4-33cca9398cd4 BootID:6296ac91-c673-4cdb-8918-5cf95054bcb7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:be:05:09:5c:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:be:05:09:5c:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:95:2d:7e:ec:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 20:58:36.657537 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.657527 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 20:58:36.657692 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.657661 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 20:58:36.661223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.661197 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 20:58:36.661352 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.661225 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 20:58:36.662087 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.662078 2585 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 20:58:36.662119 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.662090 2585 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 20:58:36.662119 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.662097 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 20:58:36.662946 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.662936 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 20:58:36.664081 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.664072 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 20:58:36.666585 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.666567 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 20:58:36.668725 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668713 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 10 20:58:36.668850 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668841 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 20:58:36.668878 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668869 2585 kubelet.go:486] "Attempting to sync node with API server" Sep 10 20:58:36.668916 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668878 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 20:58:36.668916 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668896 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 20:58:36.668916 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668905 2585 kubelet.go:398] "Adding apiserver pod source" Sep 10 20:58:36.668916 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.668913 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 20:58:36.669266 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.669252 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 20:58:36.669330 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.669276 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 20:58:36.671640 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.671622 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 20:58:36.674394 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.674381 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 20:58:36.674448 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.674411 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 20:58:36.676553 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676538 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 20:58:36.676629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676559 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 20:58:36.676629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676568 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 20:58:36.676629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676577 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 20:58:36.676629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676590 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 20:58:36.676629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676600 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676633 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676642 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676652 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676661 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676673 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 20:58:36.676766 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.676680 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 20:58:36.677558 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.677545 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 20:58:36.677588 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.677561 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 20:58:36.681368 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.681355 2585 server.go:1268] "Started kubelet" Sep 10 20:58:36.681476 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.681436 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 20:58:36.681587 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.681486 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 20:58:36.681587 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.681583 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 20:58:36.682187 ip-10-0-130-192 systemd[1]: Started Kubernetes Kubelet. Sep 10 20:58:36.683220 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.683058 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 20:58:36.683798 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.683785 2585 server.go:310] "Adding debug handlers to kubelet server" Sep 10 20:58:36.686586 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.686569 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 20:58:36.687991 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.687975 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 20:58:36.688044 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.687994 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 20:58:36.688543 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.688520 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 20:58:36.689495 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.688199 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-192.ec2.internal.18d4113f90b51e44 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-192.ec2.internal,UID:ip-10-0-130-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-192.ec2.internal,},FirstTimestamp:2026-09-10 20:58:36.681330244 +0000 UTC m=+0.389624622,LastTimestamp:2026-09-10 20:58:36.681330244 +0000 UTC m=+0.389624622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-192.ec2.internal,}" Sep 10 20:58:36.690989 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.690973 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 20:58:36.691676 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.691658 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 20:58:36.691834 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.691813 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:36.692379 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692362 2585 factory.go:55] Registering systemd factory Sep 10 20:58:36.692438 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692403 2585 factory.go:223] Registration of the systemd container factory successfully Sep 10 20:58:36.692496 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692482 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 20:58:36.692540 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692485 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 20:58:36.692540 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692509 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 20:58:36.692636 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692569 2585 reconstruct.go:97] "Volume reconstruction finished" Sep 10 20:58:36.692636 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692579 2585 reconciler.go:29] "Reconciler: start to sync state" Sep 10 20:58:36.692691 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692648 2585 factory.go:153] Registering CRI-O factory Sep 10 20:58:36.692691 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692661 2585 factory.go:223] Registration of the crio container factory successfully Sep 10 20:58:36.692740 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692707 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 20:58:36.692740 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692724 2585 factory.go:103] Registering Raw factory Sep 10 20:58:36.692740 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.692739 2585 manager.go:1196] Started watching for new ooms in manager Sep 10 20:58:36.693141 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.693128 2585 manager.go:349] Starting recovery of all containers Sep 10 20:58:36.699024 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.696919 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jpcxq" Sep 10 20:58:36.704400 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.704383 2585 manager.go:354] Recovery completed Sep 10 20:58:36.705550 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.705528 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jpcxq" Sep 10 20:58:36.706024 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.705974 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 20:58:36.706912 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.706883 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 20:58:36.708739 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.708726 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.711222 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711205 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.711285 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711234 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.711285 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711246 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.711784 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711771 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 20:58:36.711826 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711785 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 20:58:36.711826 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711800 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 10 20:58:36.711890 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.711777 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.712603 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.712585 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.712689 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.712647 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.712689 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.712662 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.713258 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.713127 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-192.ec2.internal.18d4113f927d3770 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-192.ec2.internal,UID:ip-10-0-130-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-192.ec2.internal,},FirstTimestamp:2026-09-10 20:58:36.711221104 +0000 UTC m=+0.419515480,LastTimestamp:2026-09-10 20:58:36.711221104 +0000 UTC m=+0.419515480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-192.ec2.internal,}" Sep 10 20:58:36.714023 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.714010 2585 policy_none.go:49] "None policy: Start" Sep 10 20:58:36.714079 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.714027 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 20:58:36.714079 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.714036 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 20:58:36.715978 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.715967 2585 policy_none.go:47] "Start" Sep 10 20:58:36.759271 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759145 2585 manager.go:337] "Starting Device Plugin manager" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.759311 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759326 2585 server.go:87] "Starting device plugin registration server" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759555 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759566 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759681 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759845 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.759854 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.760727 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 20:58:36.769278 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.760770 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:36.809899 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.809863 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 20:58:36.811018 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.810995 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 20:58:36.811105 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.811048 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 20:58:36.811105 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.811073 2585 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 20:58:36.811204 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.811161 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 20:58:36.815301 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.815286 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:36.860515 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.860446 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.861385 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.861367 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.861467 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.861403 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.861467 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.861414 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.861467 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.861435 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.872820 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.872805 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.872894 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.872828 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-192.ec2.internal\": node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:36.902015 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.901997 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:36.911362 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.911335 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal"] Sep 10 20:58:36.911425 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.911399 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.912159 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.912144 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.912236 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.912173 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.912236 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.912184 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.914467 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.914454 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.914619 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.914594 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.914661 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.914643 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.915207 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915189 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.915280 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915218 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.915280 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915234 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.915280 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915191 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.915280 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915261 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.915421 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.915285 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.918046 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.918032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.918105 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.918057 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:58:36.918780 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.918759 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:58:36.918854 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.918793 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:58:36.918854 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.918808 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:58:36.945233 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.945211 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-192.ec2.internal\" not found" node="ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.949401 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:36.949386 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-192.ec2.internal\" not found" node="ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.993886 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.993863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203a93e7590a5afc1a736ff688dacdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-192.ec2.internal\" (UID: \"b203a93e7590a5afc1a736ff688dacdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.993964 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.993889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:36.993964 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:36.993905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.003081 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.003060 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.094258 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203a93e7590a5afc1a736ff688dacdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-192.ec2.internal\" (UID: \"b203a93e7590a5afc1a736ff688dacdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.094258 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.094463 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.094463 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094352 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.094463 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094361 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cfccd6a440378f9930b1757ce41bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal\" (UID: \"91cfccd6a440378f9930b1757ce41bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.094463 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.094363 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b203a93e7590a5afc1a736ff688dacdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-192.ec2.internal\" (UID: \"b203a93e7590a5afc1a736ff688dacdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.103360 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.103334 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.203503 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.203411 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.248323 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.248299 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.251888 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.251868 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.304040 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.304008 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.404632 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.404574 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.504731 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.504649 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.539168 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.539135 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:37.605816 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.605779 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.622154 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.622125 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 20:58:37.622313 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.622294 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 20:58:37.622361 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.622295 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 20:58:37.689116 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.689082 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:37.706545 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:37.706509 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-192.ec2.internal\" not found" Sep 10 20:58:37.711759 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.711728 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:53:37 +0000 UTC" deadline="2028-06-06 01:52:17.576815482 +0000 UTC" Sep 10 20:58:37.711759 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.711754 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15220h53m39.865064352s" Sep 10 20:58:37.773773 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.773679 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:37.791893 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.791861 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.802737 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.802713 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 20:58:37.803525 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.803509 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" Sep 10 20:58:37.816522 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.816503 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 20:58:37.858758 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:37.858726 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cfccd6a440378f9930b1757ce41bb7.slice/crio-253b2dcbbf776d38e2c9bb39c2fd419a37352e033fe35ccb2fa08475a87fa388 WatchSource:0}: Error finding container 253b2dcbbf776d38e2c9bb39c2fd419a37352e033fe35ccb2fa08475a87fa388: Status 404 returned error can't find the container with id 253b2dcbbf776d38e2c9bb39c2fd419a37352e033fe35ccb2fa08475a87fa388 Sep 10 20:58:37.859626 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:37.859590 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb203a93e7590a5afc1a736ff688dacdd.slice/crio-4d6ebc4d17098e08a3b3fb714f3654cf57e36c689071cc5cb5009128c864095d WatchSource:0}: Error finding container 4d6ebc4d17098e08a3b3fb714f3654cf57e36c689071cc5cb5009128c864095d: Status 404 returned error can't find the container with id 4d6ebc4d17098e08a3b3fb714f3654cf57e36c689071cc5cb5009128c864095d Sep 10 20:58:37.863921 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:37.863904 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 20:58:38.670556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.670503 2585 apiserver.go:52] "Watching apiserver" Sep 10 20:58:38.678158 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.678117 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 20:58:38.679341 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.679307 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal","openshift-multus/multus-2gmjj","openshift-network-diagnostics/network-check-target-wlqrb","openshift-network-operator/iptables-alerter-dfjl4","openshift-ovn-kubernetes/ovnkube-node-g68vl","kube-system/global-pull-secret-syncer-trv4s","kube-system/konnectivity-agent-dj2ws","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv","openshift-image-registry/node-ca-2gvzb","openshift-multus/multus-additional-cni-plugins-49tl9","openshift-multus/network-metrics-daemon-fkjxm","kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zng7k","openshift-dns/node-resolver-r5hf9"] Sep 10 20:58:38.682166 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.682115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.684477 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.684453 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.687155 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.686898 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:38.687155 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.687005 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:38.689701 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.689365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.691605 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.691472 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 20:58:38.692365 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.691963 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.694388 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.694366 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.694493 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.694446 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:38.696435 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.696413 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 20:58:38.696890 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.696864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.697760 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.697742 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sxn8g\"" Sep 10 20:58:38.698843 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.698821 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.699264 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.699243 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.699716 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.699697 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.701810 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.701785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.704798 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.704762 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.706341 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-lib-modules\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.706440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-tuned\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.706440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:38.706440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bvl\" (UniqueName: \"kubernetes.io/projected/c6cb0634-11f5-49f5-a205-dff9474e7a3b-kube-api-access-26bvl\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.706440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706431 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-systemd-units\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706455 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-ovn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-node-log\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706506 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-netns\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706548 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmmn\" (UniqueName: \"kubernetes.io/projected/ae84f37d-5926-44bd-9815-a986b2fc28c0-kube-api-access-hwmmn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-tmp\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706604 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8sv\" (UniqueName: \"kubernetes.io/projected/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-kube-api-access-vf8sv\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706647 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-config\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.706688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-modprobe-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706711 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysconfig\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-systemd\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706830 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-env-overrides\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovn-node-metrics-cert\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-log-socket\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-bin\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-dbus\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.706990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-systemd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707010 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6hxkt\"" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707051 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-etc-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707080 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-script-lib\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707130 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9439980b-2015-4a42-b147-01b0711cf26f-konnectivity-ca\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.707149 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-kubernetes\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-run\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-sys\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707225 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-kubelet\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-netd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707322 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9439980b-2015-4a42-b147-01b0711cf26f-agent-certs\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-conf\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707325 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-var-lib-kubelet\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.707446 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707460 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-host\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6cb0634-11f5-49f5-a205-dff9474e7a3b-iptables-alerter-script\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707525 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6cb0634-11f5-49f5-a205-dff9474e7a3b-host-slash\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707556 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-slash\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707615 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-var-lib-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.707914 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.707661 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-kubelet-config\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.708628 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.708364 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.709161 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.709061 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 20:58:38.709256 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.709164 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jp6cz\"" Sep 10 20:58:38.712373 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.712348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.714559 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.714538 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 20:58:38.714718 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.714672 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.714805 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.714717 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 20:58:38.715380 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.715360 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.715748 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.715727 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-225jt\"" Sep 10 20:58:38.716006 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.715959 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 20:58:38.716006 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.715957 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 20:58:38.719098 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.718456 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 20:58:38.719098 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.718552 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 20:58:38.719098 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.718579 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.719098 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.718839 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.719098 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.718890 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.719397 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.719240 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v479g\"" Sep 10 20:58:38.719499 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.719478 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7znqc\"" Sep 10 20:58:38.723707 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.723684 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.724006 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.723985 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 20:58:38.724099 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724082 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 20:58:38.724168 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724117 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.724248 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724228 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.724484 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724462 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.724650 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724581 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 20:58:38.724738 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.724719 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 20:58:38.728580 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.728559 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hj6np\"" Sep 10 20:58:38.728580 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.728580 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cbvg6\"" Sep 10 20:58:38.728758 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.728721 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 20:58:38.728827 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.728760 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwr4b\"" Sep 10 20:58:38.728908 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.728881 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 20:58:38.756127 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.756066 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:38.756501 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.756371 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:58:38.756501 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.756377 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 20:58:38.768907 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.768879 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tqrld" Sep 10 20:58:38.794111 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.794081 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 20:58:38.807132 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.806852 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tqrld" Sep 10 20:58:38.808299 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808272 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-log-socket\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808381 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808310 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-dbus\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.808381 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808332 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdmg\" (UniqueName: \"kubernetes.io/projected/b5526c20-190b-471e-9912-4e50d4f40937-kube-api-access-ctdmg\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.808381 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808351 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.808556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-log-socket\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808440 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-dbus\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.808556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-registration-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.808556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808556 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-kubernetes\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-run\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-var-lib-kubelet\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-host\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808615 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5526c20-190b-471e-9912-4e50d4f40937-host\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5526c20-190b-471e-9912-4e50d4f40937-serviceca\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808728 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a40174-e77b-4d74-b7d2-54a73dc950ae-tmp-dir\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-run\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-kubelet\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808805 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-var-lib-kubelet\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808806 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.808836 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-kubernetes\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2hn\" (UniqueName: \"kubernetes.io/projected/60a40174-e77b-4d74-b7d2-54a73dc950ae-kube-api-access-dr2hn\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-system-cni-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808896 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-host\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cnibin\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-kubelet\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-etc-kubernetes\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.808971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6cb0634-11f5-49f5-a205-dff9474e7a3b-host-slash\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-slash\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809037 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-binary-copy\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809071 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-slash\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809084 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6cb0634-11f5-49f5-a205-dff9474e7a3b-host-slash\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809133 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-daemon-config\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-ovn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-node-log\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.809566 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809248 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-ovn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809259 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-node-log\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809296 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmmn\" (UniqueName: \"kubernetes.io/projected/ae84f37d-5926-44bd-9815-a986b2fc28c0-kube-api-access-hwmmn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809351 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-tmp\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-cni-binary-copy\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-bin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809492 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fh8\" (UniqueName: \"kubernetes.io/projected/5746e120-4f10-4d85-a85b-bdab5a060ac5-kube-api-access-t5fh8\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-config\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysconfig\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809690 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysconfig\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809729 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-systemd\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60a40174-e77b-4d74-b7d2-54a73dc950ae-hosts-file\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-kubelet\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.809852 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809898 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 20:58:38.810450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.809909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-systemd\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.809927 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.309904745 +0000 UTC m=+3.018199127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810071 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-multus\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-bin\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810131 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-system-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810155 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-k8s-cni-cncf-io\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810160 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-bin\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810215 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-device-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810228 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-config\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810245 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-systemd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-etc-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-script-lib\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9439980b-2015-4a42-b147-01b0711cf26f-konnectivity-ca\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-sys\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.811386 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810373 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-etc-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810388 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810445 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-multus-certs\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-run-systemd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810546 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-sys\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-netd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9439980b-2015-4a42-b147-01b0711cf26f-agent-certs\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-conf\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810666 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-cni-netd\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-tuned\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6cb0634-11f5-49f5-a205-dff9474e7a3b-iptables-alerter-script\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-var-lib-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-kubelet-config\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810813 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-lib-modules\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-cnibin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qstp\" (UniqueName: \"kubernetes.io/projected/3487b180-4035-4638-b5bd-97a5dcc6501f-kube-api-access-6qstp\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.812295 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810893 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xnvf\" (UniqueName: \"kubernetes.io/projected/85647140-f29b-4aac-ba00-20c319867dcd-kube-api-access-9xnvf\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810907 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9439980b-2015-4a42-b147-01b0711cf26f-konnectivity-ca\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-sys-fs\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810972 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-var-lib-openvswitch\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.810976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bvl\" (UniqueName: \"kubernetes.io/projected/c6cb0634-11f5-49f5-a205-dff9474e7a3b-kube-api-access-26bvl\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-systemd-units\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-netns\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-conf-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811089 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811088 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovnkube-script-lib\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-netns\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8sv\" (UniqueName: \"kubernetes.io/projected/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-kube-api-access-vf8sv\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811169 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-hostroot\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-modprobe-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811225 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811246 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-kubelet-config\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:38.813223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-env-overrides\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811292 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-sysctl-conf\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811314 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovn-node-metrics-cert\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbmt\" (UniqueName: \"kubernetes.io/projected/04fa3508-acd2-42fa-8dd4-252b7c001ff9-kube-api-access-5dbmt\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-socket-dir-parent\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-socket-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-lib-modules\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae84f37d-5926-44bd-9815-a986b2fc28c0-env-overrides\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811755 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-host-run-netns\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.811170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae84f37d-5926-44bd-9815-a986b2fc28c0-systemd-units\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.812076 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-modprobe-d\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.814162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.812642 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6cb0634-11f5-49f5-a205-dff9474e7a3b-iptables-alerter-script\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.815392 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.815363 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-tmp\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.816404 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.816321 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9439980b-2015-4a42-b147-01b0711cf26f-agent-certs\") pod \"konnectivity-agent-dj2ws\" (UID: \"9439980b-2015-4a42-b147-01b0711cf26f\") " pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:38.816773 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.816749 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-etc-tuned\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.816859 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.816754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae84f37d-5926-44bd-9815-a986b2fc28c0-ovn-node-metrics-cert\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.818629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.818582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" event={"ID":"b203a93e7590a5afc1a736ff688dacdd","Type":"ContainerStarted","Data":"4d6ebc4d17098e08a3b3fb714f3654cf57e36c689071cc5cb5009128c864095d"} Sep 10 20:58:38.820459 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.820433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" event={"ID":"91cfccd6a440378f9930b1757ce41bb7","Type":"ContainerStarted","Data":"253b2dcbbf776d38e2c9bb39c2fd419a37352e033fe35ccb2fa08475a87fa388"} Sep 10 20:58:38.823443 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.823422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bvl\" (UniqueName: \"kubernetes.io/projected/c6cb0634-11f5-49f5-a205-dff9474e7a3b-kube-api-access-26bvl\") pod \"iptables-alerter-dfjl4\" (UID: \"c6cb0634-11f5-49f5-a205-dff9474e7a3b\") " pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:38.825796 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.825772 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:38.825796 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.825802 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:38.825958 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.825816 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:38.825958 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.825875 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.325856195 +0000 UTC m=+3.034150562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:38.829544 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.829520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmmn\" (UniqueName: \"kubernetes.io/projected/ae84f37d-5926-44bd-9815-a986b2fc28c0-kube-api-access-hwmmn\") pod \"ovnkube-node-g68vl\" (UID: \"ae84f37d-5926-44bd-9815-a986b2fc28c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:38.834317 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.834249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8sv\" (UniqueName: \"kubernetes.io/projected/da52ad69-c0e5-421f-b2df-8f7e0a9fb237-kube-api-access-vf8sv\") pod \"tuned-zng7k\" (UID: \"da52ad69-c0e5-421f-b2df-8f7e0a9fb237\") " pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:38.912643 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-cni-binary-copy\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-bin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fh8\" (UniqueName: \"kubernetes.io/projected/5746e120-4f10-4d85-a85b-bdab5a060ac5-kube-api-access-t5fh8\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60a40174-e77b-4d74-b7d2-54a73dc950ae-hosts-file\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-kubelet\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912780 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-multus\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.912828 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912804 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-bin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-system-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-k8s-cni-cncf-io\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912865 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/60a40174-e77b-4d74-b7d2-54a73dc950ae-hosts-file\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912926 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-device-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912976 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-system-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-multus-certs\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-cni-multus\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913038 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-device-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913081 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-cnibin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-k8s-cni-cncf-io\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qstp\" (UniqueName: \"kubernetes.io/projected/3487b180-4035-4638-b5bd-97a5dcc6501f-kube-api-access-6qstp\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.912926 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-var-lib-kubelet\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.913216 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.913196 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913237 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-cni-binary-copy\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:38.913254 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.413233821 +0000 UTC m=+3.121528187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-cni-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913323 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xnvf\" (UniqueName: \"kubernetes.io/projected/85647140-f29b-4aac-ba00-20c319867dcd-kube-api-access-9xnvf\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-sys-fs\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913357 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-cnibin\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913379 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-netns\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-conf-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913418 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-multus-certs\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913428 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913428 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913439 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-host-run-netns\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913448 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-sys-fs\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913452 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-hostroot\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-conf-dir\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.914031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913496 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-hostroot\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913500 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbmt\" (UniqueName: \"kubernetes.io/projected/04fa3508-acd2-42fa-8dd4-252b7c001ff9-kube-api-access-5dbmt\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-socket-dir-parent\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-socket-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913618 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdmg\" (UniqueName: \"kubernetes.io/projected/b5526c20-190b-471e-9912-4e50d4f40937-kube-api-access-ctdmg\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913651 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-socket-dir-parent\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913659 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913672 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-registration-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913716 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-socket-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5526c20-190b-471e-9912-4e50d4f40937-host\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-registration-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913844 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5746e120-4f10-4d85-a85b-bdab5a060ac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5526c20-190b-471e-9912-4e50d4f40937-host\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5526c20-190b-471e-9912-4e50d4f40937-serviceca\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.914866 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a40174-e77b-4d74-b7d2-54a73dc950ae-tmp-dir\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2hn\" (UniqueName: \"kubernetes.io/projected/60a40174-e77b-4d74-b7d2-54a73dc950ae-kube-api-access-dr2hn\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-system-cni-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913969 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cnibin\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.913986 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-etc-kubernetes\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-binary-copy\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-daemon-config\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914054 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cnibin\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914065 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3487b180-4035-4638-b5bd-97a5dcc6501f-etc-kubernetes\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04fa3508-acd2-42fa-8dd4-252b7c001ff9-system-cni-dir\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914213 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a40174-e77b-4d74-b7d2-54a73dc950ae-tmp-dir\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914309 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5526c20-190b-471e-9912-4e50d4f40937-serviceca\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914558 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3487b180-4035-4638-b5bd-97a5dcc6501f-multus-daemon-config\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.915437 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.914558 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04fa3508-acd2-42fa-8dd4-252b7c001ff9-cni-binary-copy\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.932794 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.932710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xnvf\" (UniqueName: \"kubernetes.io/projected/85647140-f29b-4aac-ba00-20c319867dcd-kube-api-access-9xnvf\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:38.934046 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.934010 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qstp\" (UniqueName: \"kubernetes.io/projected/3487b180-4035-4638-b5bd-97a5dcc6501f-kube-api-access-6qstp\") pod \"multus-2gmjj\" (UID: \"3487b180-4035-4638-b5bd-97a5dcc6501f\") " pod="openshift-multus/multus-2gmjj" Sep 10 20:58:38.934201 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.934166 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdmg\" (UniqueName: \"kubernetes.io/projected/b5526c20-190b-471e-9912-4e50d4f40937-kube-api-access-ctdmg\") pod \"node-ca-2gvzb\" (UID: \"b5526c20-190b-471e-9912-4e50d4f40937\") " pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:38.937338 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.937312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2hn\" (UniqueName: \"kubernetes.io/projected/60a40174-e77b-4d74-b7d2-54a73dc950ae-kube-api-access-dr2hn\") pod \"node-resolver-r5hf9\" (UID: \"60a40174-e77b-4d74-b7d2-54a73dc950ae\") " pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:38.940745 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.940721 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbmt\" (UniqueName: \"kubernetes.io/projected/04fa3508-acd2-42fa-8dd4-252b7c001ff9-kube-api-access-5dbmt\") pod \"multus-additional-cni-plugins-49tl9\" (UID: \"04fa3508-acd2-42fa-8dd4-252b7c001ff9\") " pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:38.942318 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.942296 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fh8\" (UniqueName: \"kubernetes.io/projected/5746e120-4f10-4d85-a85b-bdab5a060ac5-kube-api-access-t5fh8\") pod \"aws-ebs-csi-driver-node-bqdcv\" (UID: \"5746e120-4f10-4d85-a85b-bdab5a060ac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:38.995483 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:38.995444 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:39.005170 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.005149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zng7k" Sep 10 20:58:39.018752 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.018731 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dfjl4" Sep 10 20:58:39.025767 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.025748 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:58:39.033495 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.033478 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gvzb" Sep 10 20:58:39.041731 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.041716 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5hf9" Sep 10 20:58:39.050150 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.050129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gmjj" Sep 10 20:58:39.058019 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.057998 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-49tl9" Sep 10 20:58:39.064973 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.064949 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" Sep 10 20:58:39.316790 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.316700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:39.316938 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.316804 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:39.316938 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.316884 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.316858589 +0000 UTC m=+4.025152956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:39.418070 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.418024 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:39.418255 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.418084 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:39.418255 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418205 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:39.418255 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418205 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:39.418255 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418223 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:39.418255 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418240 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:39.418535 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418284 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.418265876 +0000 UTC m=+4.126560242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:39.418535 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.418302 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.418293065 +0000 UTC m=+4.126587431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:39.810205 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.810082 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:39 +0000 UTC" deadline="2028-05-09 23:42:36.052150341 +0000 UTC" Sep 10 20:58:39.810205 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.810144 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14570h43m56.24200982s" Sep 10 20:58:39.811345 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:39.811320 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:39.811486 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:39.811461 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:39.862724 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.862697 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a40174_e77b_4d74_b7d2_54a73dc950ae.slice/crio-63e4e09c538f2523c8922c5ee78af68939fc7abddcfe1b571aa4170f65eff61b WatchSource:0}: Error finding container 63e4e09c538f2523c8922c5ee78af68939fc7abddcfe1b571aa4170f65eff61b: Status 404 returned error can't find the container with id 63e4e09c538f2523c8922c5ee78af68939fc7abddcfe1b571aa4170f65eff61b Sep 10 20:58:39.872022 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.871995 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae84f37d_5926_44bd_9815_a986b2fc28c0.slice/crio-5ef2641b09a8bc8c8304a312ae0b5b1729486cae9c2eacbc6c925bef2e1a4f14 WatchSource:0}: Error finding container 5ef2641b09a8bc8c8304a312ae0b5b1729486cae9c2eacbc6c925bef2e1a4f14: Status 404 returned error can't find the container with id 5ef2641b09a8bc8c8304a312ae0b5b1729486cae9c2eacbc6c925bef2e1a4f14 Sep 10 20:58:39.873103 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.873061 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5746e120_4f10_4d85_a85b_bdab5a060ac5.slice/crio-5482b889bb9ee0c22a951c08187243dc6c69adc8e090c54cedcb0129556198e5 WatchSource:0}: Error finding container 5482b889bb9ee0c22a951c08187243dc6c69adc8e090c54cedcb0129556198e5: Status 404 returned error can't find the container with id 5482b889bb9ee0c22a951c08187243dc6c69adc8e090c54cedcb0129556198e5 Sep 10 20:58:39.874282 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.874189 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fa3508_acd2_42fa_8dd4_252b7c001ff9.slice/crio-a2ee1bee9a3c75f6c8fa46f31004379e366328597625b60e07ad697976b7f34f WatchSource:0}: Error finding container a2ee1bee9a3c75f6c8fa46f31004379e366328597625b60e07ad697976b7f34f: Status 404 returned error can't find the container with id a2ee1bee9a3c75f6c8fa46f31004379e366328597625b60e07ad697976b7f34f Sep 10 20:58:39.875428 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.875317 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9439980b_2015_4a42_b147_01b0711cf26f.slice/crio-fd471aa04b6277a112c9a00fb3c9bbccc6468a6e50de76680822a978e7269b48 WatchSource:0}: Error finding container fd471aa04b6277a112c9a00fb3c9bbccc6468a6e50de76680822a978e7269b48: Status 404 returned error can't find the container with id fd471aa04b6277a112c9a00fb3c9bbccc6468a6e50de76680822a978e7269b48 Sep 10 20:58:39.898790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.898772 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3487b180_4035_4638_b5bd_97a5dcc6501f.slice/crio-1070589f0b11a11234bd2275604a26442a8fa4dfcba88c7d99cf5569b71cdd13 WatchSource:0}: Error finding container 1070589f0b11a11234bd2275604a26442a8fa4dfcba88c7d99cf5569b71cdd13: Status 404 returned error can't find the container with id 1070589f0b11a11234bd2275604a26442a8fa4dfcba88c7d99cf5569b71cdd13 Sep 10 20:58:39.899998 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.899973 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda52ad69_c0e5_421f_b2df_8f7e0a9fb237.slice/crio-5bdbadd508ebf9a29d37f5765769312f5ad821a4ae2d9a3ba89bdd1059fc39a4 WatchSource:0}: Error finding container 5bdbadd508ebf9a29d37f5765769312f5ad821a4ae2d9a3ba89bdd1059fc39a4: Status 404 returned error can't find the container with id 5bdbadd508ebf9a29d37f5765769312f5ad821a4ae2d9a3ba89bdd1059fc39a4 Sep 10 20:58:39.900731 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.900674 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cb0634_11f5_49f5_a205_dff9474e7a3b.slice/crio-3d2096219cf44674891e8957a8cda927ed13d5c64081a7bd04573af028911c97 WatchSource:0}: Error finding container 3d2096219cf44674891e8957a8cda927ed13d5c64081a7bd04573af028911c97: Status 404 returned error can't find the container with id 3d2096219cf44674891e8957a8cda927ed13d5c64081a7bd04573af028911c97 Sep 10 20:58:39.901598 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:58:39.901575 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5526c20_190b_471e_9912_4e50d4f40937.slice/crio-ec803de296415f31528b2c91d1a43e5c69f1a968c2f747b08d8e676a93a9bdd9 WatchSource:0}: Error finding container ec803de296415f31528b2c91d1a43e5c69f1a968c2f747b08d8e676a93a9bdd9: Status 404 returned error can't find the container with id ec803de296415f31528b2c91d1a43e5c69f1a968c2f747b08d8e676a93a9bdd9 Sep 10 20:58:40.325544 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.325266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:40.325755 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.325427 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:40.325755 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.325674 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.325652901 +0000 UTC m=+6.033947267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:40.426494 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.426411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:40.426494 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.426476 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:40.426790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.426633 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:40.426790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.426677 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:40.426790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.426691 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:40.426790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.426755 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.426735023 +0000 UTC m=+6.135029392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:40.427256 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.427176 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:40.427256 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.427234 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.42721794 +0000 UTC m=+6.135512309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:40.547663 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.547628 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:58:40.811183 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.811043 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:39 +0000 UTC" deadline="2028-04-20 23:08:00.638489664 +0000 UTC" Sep 10 20:58:40.811183 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.811095 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14114h9m19.827399807s" Sep 10 20:58:40.811972 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.811312 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:40.811972 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.811443 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:40.811972 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.811874 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:40.812135 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:40.811974 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:40.837385 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.837315 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gmjj" event={"ID":"3487b180-4035-4638-b5bd-97a5dcc6501f","Type":"ContainerStarted","Data":"1070589f0b11a11234bd2275604a26442a8fa4dfcba88c7d99cf5569b71cdd13"} Sep 10 20:58:40.851444 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.851407 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerStarted","Data":"a2ee1bee9a3c75f6c8fa46f31004379e366328597625b60e07ad697976b7f34f"} Sep 10 20:58:40.856567 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.856539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5hf9" event={"ID":"60a40174-e77b-4d74-b7d2-54a73dc950ae","Type":"ContainerStarted","Data":"63e4e09c538f2523c8922c5ee78af68939fc7abddcfe1b571aa4170f65eff61b"} Sep 10 20:58:40.876571 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.876529 2585 generic.go:356] "Generic (PLEG): container finished" podID="91cfccd6a440378f9930b1757ce41bb7" containerID="4e22528274a090154d560d6d33681e556361c39dad4f39fb8e1cc5e8bf00895b" exitCode=0 Sep 10 20:58:40.876726 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.876649 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" event={"ID":"91cfccd6a440378f9930b1757ce41bb7","Type":"ContainerDied","Data":"4e22528274a090154d560d6d33681e556361c39dad4f39fb8e1cc5e8bf00895b"} Sep 10 20:58:40.910449 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.910417 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dfjl4" event={"ID":"c6cb0634-11f5-49f5-a205-dff9474e7a3b","Type":"ContainerStarted","Data":"3d2096219cf44674891e8957a8cda927ed13d5c64081a7bd04573af028911c97"} Sep 10 20:58:40.922694 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.922661 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zng7k" event={"ID":"da52ad69-c0e5-421f-b2df-8f7e0a9fb237","Type":"ContainerStarted","Data":"5bdbadd508ebf9a29d37f5765769312f5ad821a4ae2d9a3ba89bdd1059fc39a4"} Sep 10 20:58:40.925586 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.925557 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dj2ws" event={"ID":"9439980b-2015-4a42-b147-01b0711cf26f","Type":"ContainerStarted","Data":"fd471aa04b6277a112c9a00fb3c9bbccc6468a6e50de76680822a978e7269b48"} Sep 10 20:58:40.933355 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.933325 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" event={"ID":"5746e120-4f10-4d85-a85b-bdab5a060ac5","Type":"ContainerStarted","Data":"5482b889bb9ee0c22a951c08187243dc6c69adc8e090c54cedcb0129556198e5"} Sep 10 20:58:40.949268 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.949241 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"5ef2641b09a8bc8c8304a312ae0b5b1729486cae9c2eacbc6c925bef2e1a4f14"} Sep 10 20:58:40.966669 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.966608 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" event={"ID":"b203a93e7590a5afc1a736ff688dacdd","Type":"ContainerStarted","Data":"209ea6b933dfd3d3733ba9db22a7186c1d89319f858bdefc0d7c294d766021f2"} Sep 10 20:58:40.973652 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:40.973623 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gvzb" event={"ID":"b5526c20-190b-471e-9912-4e50d4f40937","Type":"ContainerStarted","Data":"ec803de296415f31528b2c91d1a43e5c69f1a968c2f747b08d8e676a93a9bdd9"} Sep 10 20:58:41.811495 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:41.811459 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:41.812147 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:41.811614 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:41.996033 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:41.995941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" event={"ID":"91cfccd6a440378f9930b1757ce41bb7","Type":"ContainerStarted","Data":"9e89992945baf6462734e2ae678fc93e04df391ce98df11eeb9db79688a80cc4"} Sep 10 20:58:42.018642 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.018577 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-192.ec2.internal" podStartSLOduration=4.01855513 podStartE2EDuration="4.01855513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:40.992249468 +0000 UTC m=+4.700543859" watchObservedRunningTime="2026-09-10 20:58:42.01855513 +0000 UTC m=+5.726849501" Sep 10 20:58:42.350053 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.349966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:42.350206 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.350089 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:42.350206 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.350146 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.350129558 +0000 UTC m=+10.058423935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.450855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.450947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.451081 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.451141 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.451122584 +0000 UTC m=+10.159416953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.451552 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.451571 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:42.451673 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.451583 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:42.452249 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.452217 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.452195214 +0000 UTC m=+10.160489602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:42.812143 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.812067 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:42.812577 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.812199 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:42.812690 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:42.812587 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:42.812747 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:42.812694 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:43.812244 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:43.812212 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:43.812719 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:43.812351 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:44.816300 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:44.816269 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:44.816300 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:44.816299 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:44.816772 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:44.816402 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:44.816772 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:44.816493 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:45.812136 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:45.812106 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:45.812290 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:45.812232 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:46.385744 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:46.385141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:46.385744 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.385330 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:46.385744 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.385390 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:54.385372437 +0000 UTC m=+18.093666807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:46.486074 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:46.486036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:46.486241 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:46.486114 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:46.486298 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486239 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:46.486298 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486260 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:46.486298 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486292 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:46.486465 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486302 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:58:54.486282811 +0000 UTC m=+18.194577199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:46.486465 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486307 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:46.486465 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.486359 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:54.486343856 +0000 UTC m=+18.194638223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:46.815694 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:46.815472 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:46.815694 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.815590 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:46.815694 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:46.815602 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:46.815960 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:46.815713 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:47.811959 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:47.811914 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:47.812494 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:47.812072 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:48.812147 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:48.811822 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:48.812147 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:48.812015 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:48.812752 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:48.812474 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:48.812752 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:48.812600 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:49.811442 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:49.811413 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:49.811630 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:49.811543 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:50.811718 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:50.811681 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:50.812159 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:50.811810 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:50.812159 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:50.811873 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:50.812159 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:50.811990 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:51.812247 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:51.812204 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:51.812732 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:51.812356 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:52.811454 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:52.811414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:52.811653 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:52.811431 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:52.811653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:52.811559 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:52.811784 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:52.811652 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:53.811970 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:53.811936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:53.812367 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:53.812054 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:54.446134 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:54.446096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:54.446326 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.446252 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:54.446326 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.446320 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.446301104 +0000 UTC m=+34.154595471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:58:54.547349 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:54.547313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:54.547520 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:54.547396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:54.547520 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547449 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:58:54.547520 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547473 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:58:54.547520 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547487 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:54.547520 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547486 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:54.547790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547561 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.547539185 +0000 UTC m=+34.255833552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:58:54.547790 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.547585 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.547573627 +0000 UTC m=+34.255867993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:58:54.811321 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:54.811285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:54.811489 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.811424 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:54.811489 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:54.811477 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:54.811653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:54.811597 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:55.811868 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:55.811835 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:55.812306 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:55.811969 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:56.812850 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:56.812821 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:56.812850 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:56.812845 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:56.813328 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:56.812904 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:56.813328 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:56.812970 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:57.812127 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:57.812098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:57.812280 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:57.812199 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:58:58.030423 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:58.030235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" event={"ID":"5746e120-4f10-4d85-a85b-bdab5a060ac5","Type":"ContainerStarted","Data":"5fb356d1f75960bd8bee2f2f4f91c7cfb6a3bb9cf82263f745a88a503f1bed1e"} Sep 10 20:58:58.811939 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:58.811707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:58:58.812140 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:58.811781 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:58:58.812140 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:58.812059 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:58:58.812238 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:58.812155 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:58:59.032965 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.032928 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zng7k" event={"ID":"da52ad69-c0e5-421f-b2df-8f7e0a9fb237","Type":"ContainerStarted","Data":"9a3bf4ba4278c77094a50876645502da184c7213c3d5c5fde11e342c3677064e"} Sep 10 20:58:59.034483 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.034460 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dj2ws" event={"ID":"9439980b-2015-4a42-b147-01b0711cf26f","Type":"ContainerStarted","Data":"ce0e6aff0aca080cf54f0119f50502b32b7bbf47a6b985f8adeb36eecd39ea81"} Sep 10 20:58:59.037159 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037140 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 20:58:59.037466 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037448 2585 generic.go:356] "Generic (PLEG): container finished" podID="ae84f37d-5926-44bd-9815-a986b2fc28c0" containerID="ecb51e938c0984d19803e2a8a22fa1af899bc2accd8b30a4c28c85b52b9eb60a" exitCode=1 Sep 10 20:58:59.037544 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"a68a56f811440d1054380762a0dbd9e79b5889342b903bf6f60734d479b682ab"} Sep 10 20:58:59.037593 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"62741b53df69168959f8127ff18d51854f8574b4abb3340a051bd2ae2f611335"} Sep 10 20:58:59.037657 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037628 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"ae8e2b0ccb4cd28a7b7d83c42b842cd4069d006e50448319af56eff1911a0dd9"} Sep 10 20:58:59.037703 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037667 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"984b98878268cdf811c88a7dbea3281fe9e4b2bfe029c065f53495a8561f0d3a"} Sep 10 20:58:59.037756 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037704 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerDied","Data":"ecb51e938c0984d19803e2a8a22fa1af899bc2accd8b30a4c28c85b52b9eb60a"} Sep 10 20:58:59.037807 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.037753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"eb1df0535e4a796fa46feb1792c88cd2dd7e2efa8ee8cece9ee03e07b326e27c"} Sep 10 20:58:59.038965 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.038942 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gvzb" event={"ID":"b5526c20-190b-471e-9912-4e50d4f40937","Type":"ContainerStarted","Data":"9c83b00ea3f6d9f7e2b1e1f7ba0df38346f05829b5fb1da4503490970ec82f66"} Sep 10 20:58:59.040515 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.040496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gmjj" event={"ID":"3487b180-4035-4638-b5bd-97a5dcc6501f","Type":"ContainerStarted","Data":"09b565d3a31f487a07cb4e8ca430de3e1811c1731dde065fa972c7d0b3385a35"} Sep 10 20:58:59.042021 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.041999 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="516588e717e05a9ee4a96d8c3e0f71822525153f836b49e727809b7fb3cb10dc" exitCode=0 Sep 10 20:58:59.042112 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.042072 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"516588e717e05a9ee4a96d8c3e0f71822525153f836b49e727809b7fb3cb10dc"} Sep 10 20:58:59.043495 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.043464 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5hf9" event={"ID":"60a40174-e77b-4d74-b7d2-54a73dc950ae","Type":"ContainerStarted","Data":"db1e9a59962272548bd037d1fa04cf1ef5219d106d19a1b706a716751c4d24ba"} Sep 10 20:58:59.066332 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.066292 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-192.ec2.internal" podStartSLOduration=21.066280125 podStartE2EDuration="21.066280125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:42.020015534 +0000 UTC m=+5.728309924" watchObservedRunningTime="2026-09-10 20:58:59.066280125 +0000 UTC m=+22.774574511" Sep 10 20:58:59.186274 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.186190 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dj2ws" podStartSLOduration=4.205432927 podStartE2EDuration="22.186177323s" totalImagesPullingTime="17.980744396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.185824012 +0000 UTC m=+22.894118398" watchObservedRunningTime="2026-09-10 20:58:59.186177323 +0000 UTC m=+22.894471709" Sep 10 20:58:59.186450 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.186431 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zng7k" podStartSLOduration=4.19763422 podStartE2EDuration="22.186426203s" totalImagesPullingTime="17.988791983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.069313474 +0000 UTC m=+22.777607871" watchObservedRunningTime="2026-09-10 20:58:59.186426203 +0000 UTC m=+22.894720586" Sep 10 20:58:59.242013 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.241985 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:59.242448 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.242431 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:58:59.300871 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.300834 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2gvzb" podStartSLOduration=4.327459384 podStartE2EDuration="22.300817852s" totalImagesPullingTime="17.973358468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.254116466 +0000 UTC m=+22.962410851" watchObservedRunningTime="2026-09-10 20:58:59.300817852 +0000 UTC m=+23.009112237" Sep 10 20:58:59.338752 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.338702 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r5hf9" podStartSLOduration=4.306523173 podStartE2EDuration="22.33868131s" totalImagesPullingTime="18.032158137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.301129637 +0000 UTC m=+23.009424023" watchObservedRunningTime="2026-09-10 20:58:59.33868131 +0000 UTC m=+23.046975697" Sep 10 20:58:59.393559 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.393515 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2gmjj" podStartSLOduration=3.75768185 podStartE2EDuration="22.393502039s" totalImagesPullingTime="18.635820189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.338962969 +0000 UTC m=+23.047257347" watchObservedRunningTime="2026-09-10 20:58:59.393502039 +0000 UTC m=+23.101796425" Sep 10 20:58:59.419665 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.419642 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 20:58:59.774120 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.774020 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:58:59.419661056Z","UUID":"47ab9b69-628f-4dc4-bd43-e0cae97a92a3","Handler":null,"Name":"","Endpoint":""} Sep 10 20:58:59.775905 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.775879 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 20:58:59.776030 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.775914 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 20:58:59.812124 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:58:59.812092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:58:59.812259 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:58:59.812219 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:00.048371 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.048287 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dfjl4" event={"ID":"c6cb0634-11f5-49f5-a205-dff9474e7a3b","Type":"ContainerStarted","Data":"b7a56385730be7507a92db7aa1822730603242fdac0b23efa75ffba3fea33225"} Sep 10 20:59:00.050744 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.050717 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" event={"ID":"5746e120-4f10-4d85-a85b-bdab5a060ac5","Type":"ContainerStarted","Data":"6a2d0f4bbee144090d45d891588802696169beff8f8dcae60e82a21cc629aafb"} Sep 10 20:59:00.051235 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.051212 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:59:00.052115 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.052098 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dj2ws" Sep 10 20:59:00.107161 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.107117 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dfjl4" podStartSLOduration=5.133819485 podStartE2EDuration="23.107102742s" totalImagesPullingTime="17.973283257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:00.074703197 +0000 UTC m=+23.782997583" watchObservedRunningTime="2026-09-10 20:59:00.107102742 +0000 UTC m=+23.815397128" Sep 10 20:59:00.811961 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.811685 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:00.812056 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:00.811694 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:00.812113 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:00.812052 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:00.812195 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:00.812169 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:01.055724 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:01.055688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" event={"ID":"5746e120-4f10-4d85-a85b-bdab5a060ac5","Type":"ContainerStarted","Data":"cec96d5ff8ee79c91f855aad65ceb38dfb1944b581959942cd57307489c6c892"} Sep 10 20:59:01.059389 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:01.059368 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 20:59:01.059920 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:01.059892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"76941318deff85e300e0c05f5a0fd0e32d3ed00fd659f60108f79d30db88f888"} Sep 10 20:59:01.811576 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:01.811536 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:01.811794 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:01.811662 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:02.812070 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:02.812039 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:02.812070 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:02.812076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:02.812584 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:02.812152 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:02.812584 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:02.812268 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:03.069750 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.069603 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 20:59:03.070127 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.070044 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"2cd1bb8e0ddf2c11ce1ba07f7c6a65bce4cb40efbd35a0c92c0465ea72e6e6f8"} Sep 10 20:59:03.070838 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.070386 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:03.070838 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.070468 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:03.070838 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.070601 2585 scope.go:117] "RemoveContainer" containerID="ecb51e938c0984d19803e2a8a22fa1af899bc2accd8b30a4c28c85b52b9eb60a" Sep 10 20:59:03.086573 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.086550 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:03.086882 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.086869 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:03.124755 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.124538 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bqdcv" podStartSLOduration=5.785582986 podStartE2EDuration="26.12449986s" totalImagesPullingTime="20.338916874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:01.085740016 +0000 UTC m=+24.794034395" watchObservedRunningTime="2026-09-10 20:59:03.12449986 +0000 UTC m=+26.832794243" Sep 10 20:59:03.811875 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:03.811843 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:03.812063 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:03.811952 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:04.078196 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.078136 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 20:59:04.078705 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.078674 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" event={"ID":"ae84f37d-5926-44bd-9815-a986b2fc28c0","Type":"ContainerStarted","Data":"47b8edf1b0eb0317fff1f9b133ab94d3005e5be30172ef6367dc8016cfa8e7b1"} Sep 10 20:59:04.078705 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.078691 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 20:59:04.080495 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.080468 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="46991f3c08579839c3792063655e94edb660126e55677579de8ad7c925393104" exitCode=0 Sep 10 20:59:04.080699 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.080506 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"46991f3c08579839c3792063655e94edb660126e55677579de8ad7c925393104"} Sep 10 20:59:04.120571 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.120536 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" podStartSLOduration=8.906877361 podStartE2EDuration="27.120523569s" totalImagesPullingTime="18.213646208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:04.120254893 +0000 UTC m=+27.828549280" watchObservedRunningTime="2026-09-10 20:59:04.120523569 +0000 UTC m=+27.828817954" Sep 10 20:59:04.811721 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.811688 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:04.811949 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:04.811790 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:04.811949 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:04.811847 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:04.812045 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:04.811963 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:05.091692 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.091668 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 20:59:05.160530 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.160503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkjxm"] Sep 10 20:59:05.160706 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.160680 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:05.160853 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:05.160811 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:05.165342 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.165315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trv4s"] Sep 10 20:59:05.165451 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.165414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:05.165533 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:05.165512 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:05.166038 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.166019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlqrb"] Sep 10 20:59:05.166133 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:05.166116 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:05.166221 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:05.166204 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:06.087532 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.087497 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="8df4af162cb86167daf8043fbd7caa4f0a1c73124d501fb4f66b42703dabf031" exitCode=0 Sep 10 20:59:06.087532 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.087534 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"8df4af162cb86167daf8043fbd7caa4f0a1c73124d501fb4f66b42703dabf031"} Sep 10 20:59:06.173549 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.173523 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:06.814910 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.814753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:06.815060 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.814754 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:06.815060 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:06.814981 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:06.815170 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:06.814763 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:06.815170 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:06.815071 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:06.815170 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:06.815160 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:08.094080 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:08.094050 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="3d0b3b1fa363b09b0dc10b3239a8cfb3ca7fbaca51a077ada3a1d3372f9bebab" exitCode=0 Sep 10 20:59:08.094448 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:08.094109 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"3d0b3b1fa363b09b0dc10b3239a8cfb3ca7fbaca51a077ada3a1d3372f9bebab"} Sep 10 20:59:08.812349 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:08.812313 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:08.812510 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:08.812320 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:08.812510 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:08.812424 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:08.812510 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:08.812447 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:08.812699 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:08.812560 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:08.812699 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:08.812660 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:10.462548 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.462514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:10.463015 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.462679 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:59:10.463015 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.462751 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret podName:0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.462728455 +0000 UTC m=+66.171022819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret") pod "global-pull-secret-syncer-trv4s" (UID: "0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:59:10.563537 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.563504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:10.563715 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.563561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:10.563715 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563676 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:59:10.563715 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563707 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:59:10.563849 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563726 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:59:10.563849 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563738 2585 projected.go:196] Error preparing data for projected volume kube-api-access-t4vxs for pod openshift-network-diagnostics/network-check-target-wlqrb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:59:10.563849 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563745 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs podName:85647140-f29b-4aac-ba00-20c319867dcd nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.563728437 +0000 UTC m=+66.272022801 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs") pod "network-metrics-daemon-fkjxm" (UID: "85647140-f29b-4aac-ba00-20c319867dcd") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:59:10.563849 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.563785 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs podName:94843d1c-3563-4551-9ddd-a71aebaa186c nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.563769065 +0000 UTC m=+66.272063432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vxs" (UniqueName: "kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs") pod "network-check-target-wlqrb" (UID: "94843d1c-3563-4551-9ddd-a71aebaa186c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:59:10.812399 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.812373 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:10.812399 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.812395 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:10.812668 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:10.812376 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:10.812668 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.812495 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fkjxm" podUID="85647140-f29b-4aac-ba00-20c319867dcd" Sep 10 20:59:10.812668 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.812571 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlqrb" podUID="94843d1c-3563-4551-9ddd-a71aebaa186c" Sep 10 20:59:10.812668 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:10.812652 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trv4s" podUID="0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0" Sep 10 20:59:11.119043 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.118978 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-192.ec2.internal" event="NodeReady" Sep 10 20:59:11.119164 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.119121 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 20:59:11.250083 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.250048 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t6h5z"] Sep 10 20:59:11.255142 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.255116 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.261527 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.261499 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zxhqt"] Sep 10 20:59:11.264878 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.264857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.274940 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.274921 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vl2hm\"" Sep 10 20:59:11.275789 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.275767 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 20:59:11.275789 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.275782 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 20:59:11.284094 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.284055 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 20:59:11.284190 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.284143 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 20:59:11.284190 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.284149 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 20:59:11.284284 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.284255 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tkv7t\"" Sep 10 20:59:11.293589 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.293556 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t6h5z"] Sep 10 20:59:11.308816 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.308797 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zxhqt"] Sep 10 20:59:11.369588 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-config-volume\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.369588 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjcm\" (UniqueName: \"kubernetes.io/projected/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-kube-api-access-htjcm\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.369807 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369631 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.369807 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqg7\" (UniqueName: \"kubernetes.io/projected/33e80727-2e90-4091-84aa-90798cffb324-kube-api-access-cfqg7\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.369807 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33e80727-2e90-4091-84aa-90798cffb324-cert\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.369958 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.369810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-tmp-dir\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.470900 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.470855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjcm\" (UniqueName: \"kubernetes.io/projected/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-kube-api-access-htjcm\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.470916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:11.471038 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:11.471098 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:11.971078285 +0000 UTC m=+35.679372667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqg7\" (UniqueName: \"kubernetes.io/projected/33e80727-2e90-4091-84aa-90798cffb324-kube-api-access-cfqg7\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33e80727-2e90-4091-84aa-90798cffb324-cert\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-tmp-dir\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.471357 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-config-volume\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.471736 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-tmp-dir\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.471860 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.471839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-config-volume\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.476148 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.476125 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33e80727-2e90-4091-84aa-90798cffb324-cert\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.485006 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.484984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqg7\" (UniqueName: \"kubernetes.io/projected/33e80727-2e90-4091-84aa-90798cffb324-kube-api-access-cfqg7\") pod \"ingress-canary-zxhqt\" (UID: \"33e80727-2e90-4091-84aa-90798cffb324\") " pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.487318 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.487295 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjcm\" (UniqueName: \"kubernetes.io/projected/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-kube-api-access-htjcm\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.573119 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.573085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zxhqt" Sep 10 20:59:11.975787 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:11.975748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:11.975965 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:11.975895 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:11.975965 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:11.975951 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.975936977 +0000 UTC m=+36.684231341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:12.677465 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.677274 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zxhqt"] Sep 10 20:59:12.680561 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:12.680535 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e80727_2e90_4091_84aa_90798cffb324.slice/crio-35cb89fe46799525fdc51db5321154617f3053b810bef773e14a4f6a6e6aa4ab WatchSource:0}: Error finding container 35cb89fe46799525fdc51db5321154617f3053b810bef773e14a4f6a6e6aa4ab: Status 404 returned error can't find the container with id 35cb89fe46799525fdc51db5321154617f3053b810bef773e14a4f6a6e6aa4ab Sep 10 20:59:12.811456 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.811427 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:12.811456 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.811440 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:12.811587 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.811427 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:12.824355 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.824340 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 20:59:12.825139 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.825121 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 20:59:12.825229 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.825149 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 20:59:12.825229 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.825195 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 20:59:12.825229 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.825206 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-947bp\"" Sep 10 20:59:12.825363 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.825235 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qc4z\"" Sep 10 20:59:12.982338 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:12.982281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:12.982438 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:12.982424 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:12.982501 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:12.982476 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:14.982462307 +0000 UTC m=+38.690756671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:13.107650 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:13.107599 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zxhqt" event={"ID":"33e80727-2e90-4091-84aa-90798cffb324","Type":"ContainerStarted","Data":"35cb89fe46799525fdc51db5321154617f3053b810bef773e14a4f6a6e6aa4ab"} Sep 10 20:59:13.111595 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:13.111573 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="55e006ecaa8e89227f63d58a9bd27154e8c8e956d766fef9626577092661c8b1" exitCode=0 Sep 10 20:59:13.111699 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:13.111637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"55e006ecaa8e89227f63d58a9bd27154e8c8e956d766fef9626577092661c8b1"} Sep 10 20:59:14.118023 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.117986 2585 generic.go:356] "Generic (PLEG): container finished" podID="04fa3508-acd2-42fa-8dd4-252b7c001ff9" containerID="44154c50f4d2d4bab36533dffc47f3d27755ea504c2dabf5826ee2037544070e" exitCode=0 Sep 10 20:59:14.118792 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.118045 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerDied","Data":"44154c50f4d2d4bab36533dffc47f3d27755ea504c2dabf5826ee2037544070e"} Sep 10 20:59:14.939548 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.939495 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77"] Sep 10 20:59:14.961800 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.961774 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-x964d"] Sep 10 20:59:14.962047 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.962025 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" Sep 10 20:59:14.968844 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.968823 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fpfl2\"" Sep 10 20:59:14.968945 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.968823 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 20:59:14.968945 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.968873 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:59:14.980093 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.980072 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77"] Sep 10 20:59:14.980204 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.980173 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:14.983959 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.983942 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hp5rm\"" Sep 10 20:59:14.985014 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.984944 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 20:59:14.985300 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.985282 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 20:59:14.985562 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.985542 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 20:59:14.985562 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.985551 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 20:59:14.993306 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.993285 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-x964d"] Sep 10 20:59:14.994883 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.994865 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 20:59:14.997022 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:14.997003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:14.997145 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:14.997133 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:14.997196 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:14.997187 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:18.997169525 +0000 UTC m=+42.705463897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:15.053276 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.053251 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h"] Sep 10 20:59:15.087002 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.086985 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h"] Sep 10 20:59:15.087090 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.087057 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.094821 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.094804 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 20:59:15.094930 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.094915 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 20:59:15.095172 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.095155 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tx4fn\"" Sep 10 20:59:15.095216 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.095192 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:59:15.095603 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.095588 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 20:59:15.097423 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097405 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-tmp\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.097491 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097436 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-snapshots\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.097491 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qtk\" (UniqueName: \"kubernetes.io/projected/7d63a201-9b23-4e47-b17f-e60883195730-kube-api-access-58qtk\") pod \"volume-data-source-validator-86695b7cc7-7zl77\" (UID: \"7d63a201-9b23-4e47-b17f-e60883195730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" Sep 10 20:59:15.097603 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097541 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtqh\" (UniqueName: \"kubernetes.io/projected/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-kube-api-access-rqtqh\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.097670 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.097718 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-service-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.097718 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.097696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-serving-cert\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.123768 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.123741 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49tl9" event={"ID":"04fa3508-acd2-42fa-8dd4-252b7c001ff9","Type":"ContainerStarted","Data":"c244d8da29caa31debcd945de67de06f6f725c79e49473d991dd076529afa756"} Sep 10 20:59:15.136369 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.136341 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pf8tm"] Sep 10 20:59:15.151632 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.151601 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs"] Sep 10 20:59:15.151739 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.151725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.154408 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.154389 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5587v\"" Sep 10 20:59:15.155024 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.155006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 20:59:15.162440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.159703 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 20:59:15.162440 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.160055 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:59:15.166244 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.166225 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 20:59:15.168457 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.168441 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 20:59:15.169935 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.169920 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pf8tm"] Sep 10 20:59:15.170004 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.169994 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.182521 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.182500 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 20:59:15.182996 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.182981 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:59:15.183141 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.183125 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v628r\"" Sep 10 20:59:15.185104 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.185067 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-49tl9" podStartSLOduration=9.79751497 podStartE2EDuration="38.185056832s" totalImagesPullingTime="28.387541862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.183353356 +0000 UTC m=+38.891647785" watchObservedRunningTime="2026-09-10 20:59:15.185056832 +0000 UTC m=+38.893351265" Sep 10 20:59:15.185918 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.185905 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs"] Sep 10 20:59:15.187272 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.187256 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 20:59:15.198065 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qtk\" (UniqueName: \"kubernetes.io/projected/7d63a201-9b23-4e47-b17f-e60883195730-kube-api-access-58qtk\") pod \"volume-data-source-validator-86695b7cc7-7zl77\" (UID: \"7d63a201-9b23-4e47-b17f-e60883195730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" Sep 10 20:59:15.198065 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198056 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtqh\" (UniqueName: \"kubernetes.io/projected/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-kube-api-access-rqtqh\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198172 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a662b93e-879c-4a28-af5e-1b23552d22d2-serving-cert\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.198232 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198266 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198247 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a662b93e-879c-4a28-af5e-1b23552d22d2-config\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.198298 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-service-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198298 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-serving-cert\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198415 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198398 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-tmp\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198451 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb9s\" (UniqueName: \"kubernetes.io/projected/a662b93e-879c-4a28-af5e-1b23552d22d2-kube-api-access-kwb9s\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.198497 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-snapshots\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198792 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-tmp\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.198854 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198804 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-service-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.199135 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.198988 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-snapshots\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.199190 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.199027 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.201505 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.201478 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-serving-cert\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.217879 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.217856 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qtk\" (UniqueName: \"kubernetes.io/projected/7d63a201-9b23-4e47-b17f-e60883195730-kube-api-access-58qtk\") pod \"volume-data-source-validator-86695b7cc7-7zl77\" (UID: \"7d63a201-9b23-4e47-b17f-e60883195730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" Sep 10 20:59:15.239651 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.239600 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtqh\" (UniqueName: \"kubernetes.io/projected/4c728a21-5a9c-49d4-8e05-dbdbc6ded43a-kube-api-access-rqtqh\") pod \"insights-operator-5844b856db-x964d\" (UID: \"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a\") " pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.270281 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.270259 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" Sep 10 20:59:15.290106 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.290088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-x964d" Sep 10 20:59:15.299202 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb9s\" (UniqueName: \"kubernetes.io/projected/a662b93e-879c-4a28-af5e-1b23552d22d2-kube-api-access-kwb9s\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.299269 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299218 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-trusted-ca\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.299269 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299244 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2xv\" (UniqueName: \"kubernetes.io/projected/2555a5e5-54fd-48e0-907a-4aef7afcb35c-kube-api-access-8w2xv\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.299338 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299309 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.299378 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv26\" (UniqueName: \"kubernetes.io/projected/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-kube-api-access-kgv26\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.299429 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299386 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-config\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.299498 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299479 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a662b93e-879c-4a28-af5e-1b23552d22d2-serving-cert\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.299557 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2555a5e5-54fd-48e0-907a-4aef7afcb35c-serving-cert\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.299629 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.299561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a662b93e-879c-4a28-af5e-1b23552d22d2-config\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.300057 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.300039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a662b93e-879c-4a28-af5e-1b23552d22d2-config\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.301394 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.301373 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a662b93e-879c-4a28-af5e-1b23552d22d2-serving-cert\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.310851 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.310832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb9s\" (UniqueName: \"kubernetes.io/projected/a662b93e-879c-4a28-af5e-1b23552d22d2-kube-api-access-kwb9s\") pod \"service-ca-operator-cf7c776b7-nph6h\" (UID: \"a662b93e-879c-4a28-af5e-1b23552d22d2\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.394759 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.394727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" Sep 10 20:59:15.399979 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.399955 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-trusted-ca\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.400085 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.399994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2xv\" (UniqueName: \"kubernetes.io/projected/2555a5e5-54fd-48e0-907a-4aef7afcb35c-kube-api-access-8w2xv\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.400085 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.400028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.400085 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.400067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv26\" (UniqueName: \"kubernetes.io/projected/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-kube-api-access-kgv26\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.400229 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.400092 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-config\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.400229 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.400145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2555a5e5-54fd-48e0-907a-4aef7afcb35c-serving-cert\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.400229 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:15.400200 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 20:59:15.400368 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:15.400266 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls podName:5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de nodeName:}" failed. No retries permitted until 2026-09-10 20:59:15.900245924 +0000 UTC m=+39.608540291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnzzs" (UID: "5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de") : secret "samples-operator-tls" not found Sep 10 20:59:15.401037 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.400997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-trusted-ca\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.401037 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.401020 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2555a5e5-54fd-48e0-907a-4aef7afcb35c-config\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.403035 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.403013 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2555a5e5-54fd-48e0-907a-4aef7afcb35c-serving-cert\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.419289 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.419266 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2xv\" (UniqueName: \"kubernetes.io/projected/2555a5e5-54fd-48e0-907a-4aef7afcb35c-kube-api-access-8w2xv\") pod \"console-operator-696884fb9c-pf8tm\" (UID: \"2555a5e5-54fd-48e0-907a-4aef7afcb35c\") " pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.419492 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.419472 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv26\" (UniqueName: \"kubernetes.io/projected/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-kube-api-access-kgv26\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.462322 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.462041 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:15.562680 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.562648 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77"] Sep 10 20:59:15.566892 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:15.566863 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d63a201_9b23_4e47_b17f_e60883195730.slice/crio-e76c3955cb60850f882dc94ce28fb06a9459253522e62be2781f4fc6d883d434 WatchSource:0}: Error finding container e76c3955cb60850f882dc94ce28fb06a9459253522e62be2781f4fc6d883d434: Status 404 returned error can't find the container with id e76c3955cb60850f882dc94ce28fb06a9459253522e62be2781f4fc6d883d434 Sep 10 20:59:15.597247 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.597103 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-x964d"] Sep 10 20:59:15.599790 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:15.599764 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c728a21_5a9c_49d4_8e05_dbdbc6ded43a.slice/crio-9bcfdb43a0f91e69c9a01ad0b36438ba4c518f20ec15e25e5b4b7b2c51c4f372 WatchSource:0}: Error finding container 9bcfdb43a0f91e69c9a01ad0b36438ba4c518f20ec15e25e5b4b7b2c51c4f372: Status 404 returned error can't find the container with id 9bcfdb43a0f91e69c9a01ad0b36438ba4c518f20ec15e25e5b4b7b2c51c4f372 Sep 10 20:59:15.695344 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.695315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h"] Sep 10 20:59:15.698074 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:15.698050 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda662b93e_879c_4a28_af5e_1b23552d22d2.slice/crio-4149e4256d2e144aea25cd7d3d671effb4ec20d37fa43a5fb3f4aa37531a1a96 WatchSource:0}: Error finding container 4149e4256d2e144aea25cd7d3d671effb4ec20d37fa43a5fb3f4aa37531a1a96: Status 404 returned error can't find the container with id 4149e4256d2e144aea25cd7d3d671effb4ec20d37fa43a5fb3f4aa37531a1a96 Sep 10 20:59:15.756031 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.756004 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-pf8tm"] Sep 10 20:59:15.759042 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:15.759014 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2555a5e5_54fd_48e0_907a_4aef7afcb35c.slice/crio-9e4b7d9a560386addab636fc13e0030832bbeb92a864760d4bbcead0ea446e0d WatchSource:0}: Error finding container 9e4b7d9a560386addab636fc13e0030832bbeb92a864760d4bbcead0ea446e0d: Status 404 returned error can't find the container with id 9e4b7d9a560386addab636fc13e0030832bbeb92a864760d4bbcead0ea446e0d Sep 10 20:59:15.904549 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:15.904512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:15.904735 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:15.904661 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 20:59:15.904813 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:15.904736 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls podName:5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de nodeName:}" failed. No retries permitted until 2026-09-10 20:59:16.904714563 +0000 UTC m=+40.613008927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnzzs" (UID: "5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de") : secret "samples-operator-tls" not found Sep 10 20:59:16.127648 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.127602 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zxhqt" event={"ID":"33e80727-2e90-4091-84aa-90798cffb324","Type":"ContainerStarted","Data":"685e67cbaeee4a70afd5677e16fc1cb638b847d1ec524a0db1cd9e706802c872"} Sep 10 20:59:16.128815 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.128794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x964d" event={"ID":"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a","Type":"ContainerStarted","Data":"9bcfdb43a0f91e69c9a01ad0b36438ba4c518f20ec15e25e5b4b7b2c51c4f372"} Sep 10 20:59:16.129807 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.129786 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" event={"ID":"2555a5e5-54fd-48e0-907a-4aef7afcb35c","Type":"ContainerStarted","Data":"9e4b7d9a560386addab636fc13e0030832bbeb92a864760d4bbcead0ea446e0d"} Sep 10 20:59:16.130822 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.130804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" event={"ID":"a662b93e-879c-4a28-af5e-1b23552d22d2","Type":"ContainerStarted","Data":"4149e4256d2e144aea25cd7d3d671effb4ec20d37fa43a5fb3f4aa37531a1a96"} Sep 10 20:59:16.131819 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.131800 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" event={"ID":"7d63a201-9b23-4e47-b17f-e60883195730","Type":"ContainerStarted","Data":"e76c3955cb60850f882dc94ce28fb06a9459253522e62be2781f4fc6d883d434"} Sep 10 20:59:16.153788 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.153743 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zxhqt" podStartSLOduration=2.5415281629999997 podStartE2EDuration="5.153729117s" totalImagesPullingTime="2.612200954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:16.153125148 +0000 UTC m=+39.861419535" watchObservedRunningTime="2026-09-10 20:59:16.153729117 +0000 UTC m=+39.862023493" Sep 10 20:59:16.918464 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:16.918289 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:16.918674 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:16.918572 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 20:59:16.918674 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:16.918659 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls podName:5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de nodeName:}" failed. No retries permitted until 2026-09-10 20:59:18.918638953 +0000 UTC m=+42.626933331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnzzs" (UID: "5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de") : secret "samples-operator-tls" not found Sep 10 20:59:18.935576 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:18.935538 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:18.936001 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:18.935726 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 20:59:18.936001 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:18.935799 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls podName:5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de nodeName:}" failed. No retries permitted until 2026-09-10 20:59:22.935777542 +0000 UTC m=+46.644071906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnzzs" (UID: "5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de") : secret "samples-operator-tls" not found Sep 10 20:59:19.036260 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:19.036218 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:19.036437 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:19.036395 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:19.036502 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:19.036473 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:27.036451869 +0000 UTC m=+50.744746242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:20.144555 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.144515 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x964d" event={"ID":"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a","Type":"ContainerStarted","Data":"e45c3fd1d5ed99311523b34d301c9983dbd07072673816daf6014ad12e38af5d"} Sep 10 20:59:20.146290 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.146267 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/0.log" Sep 10 20:59:20.146483 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.146309 2585 generic.go:356] "Generic (PLEG): container finished" podID="2555a5e5-54fd-48e0-907a-4aef7afcb35c" containerID="ac6d519081dc991b094ac1294ac1352db2e52e940280ccf4947ac41fbca0f21e" exitCode=255 Sep 10 20:59:20.146483 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.146379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" event={"ID":"2555a5e5-54fd-48e0-907a-4aef7afcb35c","Type":"ContainerDied","Data":"ac6d519081dc991b094ac1294ac1352db2e52e940280ccf4947ac41fbca0f21e"} Sep 10 20:59:20.146591 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.146571 2585 scope.go:117] "RemoveContainer" containerID="ac6d519081dc991b094ac1294ac1352db2e52e940280ccf4947ac41fbca0f21e" Sep 10 20:59:20.147996 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.147963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" event={"ID":"a662b93e-879c-4a28-af5e-1b23552d22d2","Type":"ContainerStarted","Data":"417c71ccee6ca721bab08cf38d07e013c691e73b6f46ec2e18890162db94009f"} Sep 10 20:59:20.149665 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.149644 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" event={"ID":"7d63a201-9b23-4e47-b17f-e60883195730","Type":"ContainerStarted","Data":"09f4af1566e1adb51c54d924e68ecd6a0b2214e9c77dd85315f7c001ac087704"} Sep 10 20:59:20.167147 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.167103 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-x964d" podStartSLOduration=2.077761106 podStartE2EDuration="6.167086045s" totalImagesPullingTime="4.089324939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:20.165505758 +0000 UTC m=+43.873800146" watchObservedRunningTime="2026-09-10 20:59:20.167086045 +0000 UTC m=+43.875380435" Sep 10 20:59:20.184713 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.184670 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" podStartSLOduration=1.193585176 podStartE2EDuration="5.184657374s" totalImagesPullingTime="3.991072198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:20.183665737 +0000 UTC m=+43.891960118" watchObservedRunningTime="2026-09-10 20:59:20.184657374 +0000 UTC m=+43.892951766" Sep 10 20:59:20.206671 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:20.206632 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7zl77" podStartSLOduration=2.085744553 podStartE2EDuration="6.206601461s" totalImagesPullingTime="4.120856908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:20.206053594 +0000 UTC m=+43.914347982" watchObservedRunningTime="2026-09-10 20:59:20.206601461 +0000 UTC m=+43.914895846" Sep 10 20:59:21.155317 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.155293 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 20:59:21.155819 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.155801 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/0.log" Sep 10 20:59:21.155907 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.155837 2585 generic.go:356] "Generic (PLEG): container finished" podID="2555a5e5-54fd-48e0-907a-4aef7afcb35c" containerID="09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969" exitCode=255 Sep 10 20:59:21.155985 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.155951 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" event={"ID":"2555a5e5-54fd-48e0-907a-4aef7afcb35c","Type":"ContainerDied","Data":"09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969"} Sep 10 20:59:21.156113 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.156023 2585 scope.go:117] "RemoveContainer" containerID="ac6d519081dc991b094ac1294ac1352db2e52e940280ccf4947ac41fbca0f21e" Sep 10 20:59:21.156370 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.156177 2585 scope.go:117] "RemoveContainer" containerID="09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969" Sep 10 20:59:21.156370 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:21.156359 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pf8tm_openshift-console-operator(2555a5e5-54fd-48e0-907a-4aef7afcb35c)\"" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" podUID="2555a5e5-54fd-48e0-907a-4aef7afcb35c" Sep 10 20:59:21.243854 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.243824 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kbf6p"] Sep 10 20:59:21.266510 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.266486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.269601 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.269573 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbf6p"] Sep 10 20:59:21.271044 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.271020 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 20:59:21.271974 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.271952 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7vxwx\"" Sep 10 20:59:21.272080 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.272046 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 20:59:21.456032 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.455969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.456032 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.455999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6abe2e34-0083-40d6-bd26-12d10effbbe0-crio-socket\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.456032 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.456021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.456223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.456111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6abe2e34-0083-40d6-bd26-12d10effbbe0-data-volume\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.456223 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.456134 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mpj\" (UniqueName: \"kubernetes.io/projected/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-api-access-n6mpj\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556398 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556368 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556532 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6abe2e34-0083-40d6-bd26-12d10effbbe0-data-volume\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556532 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mpj\" (UniqueName: \"kubernetes.io/projected/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-api-access-n6mpj\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556532 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:21.556476 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:59:21.556532 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556755 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:21.556539 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:22.056524559 +0000 UTC m=+45.764818922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : configmap "kube-rbac-proxy" not found Sep 10 20:59:21.556755 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6abe2e34-0083-40d6-bd26-12d10effbbe0-crio-socket\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556755 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:21.556593 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:21.556755 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6abe2e34-0083-40d6-bd26-12d10effbbe0-crio-socket\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.556755 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:21.556671 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:22.056654573 +0000 UTC m=+45.764948954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:21.556959 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.556795 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6abe2e34-0083-40d6-bd26-12d10effbbe0-data-volume\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:21.565514 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:21.565495 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mpj\" (UniqueName: \"kubernetes.io/projected/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-api-access-n6mpj\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:22.059198 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:22.059159 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:22.059198 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:22.059203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:22.059443 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.059311 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:22.059443 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.059328 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:59:22.059443 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.059373 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:23.059357862 +0000 UTC m=+46.767652227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:22.059443 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.059388 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:23.059382685 +0000 UTC m=+46.767677049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : configmap "kube-rbac-proxy" not found Sep 10 20:59:22.160877 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:22.160851 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 20:59:22.161263 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:22.161174 2585 scope.go:117] "RemoveContainer" containerID="09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969" Sep 10 20:59:22.161342 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.161322 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pf8tm_openshift-console-operator(2555a5e5-54fd-48e0-907a-4aef7afcb35c)\"" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" podUID="2555a5e5-54fd-48e0-907a-4aef7afcb35c" Sep 10 20:59:22.965335 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:22.965304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:22.965447 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.965419 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 20:59:22.965496 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:22.965479 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls podName:5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de nodeName:}" failed. No retries permitted until 2026-09-10 20:59:30.965464559 +0000 UTC m=+54.673758922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnzzs" (UID: "5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de") : secret "samples-operator-tls" not found Sep 10 20:59:23.066389 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.066365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:23.066478 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.066397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:23.066529 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.066510 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:23.066589 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.066580 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:25.066564739 +0000 UTC m=+48.774859117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:23.066648 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.066517 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:59:23.066681 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.066658 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:25.066647022 +0000 UTC m=+48.774941387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : configmap "kube-rbac-proxy" not found Sep 10 20:59:23.175513 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.175492 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5hf9_60a40174-e77b-4d74-b7d2-54a73dc950ae/dns-node-resolver/0.log" Sep 10 20:59:23.552826 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.552797 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j"] Sep 10 20:59:23.585926 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.585903 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j"] Sep 10 20:59:23.586048 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.585990 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.592579 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.592556 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 20:59:23.593595 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.593578 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lr75n\"" Sep 10 20:59:23.593719 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.593626 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 20:59:23.671714 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.671681 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.671843 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.671736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ec021-c528-4678-be74-66f8da1edd93-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.772988 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.772949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.773162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.773008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ec021-c528-4678-be74-66f8da1edd93-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.773162 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.773111 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:59:23.773246 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:23.773191 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert podName:9c8ec021-c528-4678-be74-66f8da1edd93 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:24.273175165 +0000 UTC m=+47.981469529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vjq5j" (UID: "9c8ec021-c528-4678-be74-66f8da1edd93") : secret "networking-console-plugin-cert" not found Sep 10 20:59:23.773662 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.773646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ec021-c528-4678-be74-66f8da1edd93-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:23.980400 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:23.980334 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gvzb_b5526c20-190b-471e-9912-4e50d4f40937/node-ca/0.log" Sep 10 20:59:24.276070 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:24.276038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:24.276442 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:24.276164 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:59:24.276442 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:24.276225 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert podName:9c8ec021-c528-4678-be74-66f8da1edd93 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:25.276211819 +0000 UTC m=+48.984506183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vjq5j" (UID: "9c8ec021-c528-4678-be74-66f8da1edd93") : secret "networking-console-plugin-cert" not found Sep 10 20:59:25.082541 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.082504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:25.082541 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.082547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:25.082767 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.082675 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:25.082767 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.082711 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:59:25.082767 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.082734 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:29.082719405 +0000 UTC m=+52.791013783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:25.082882 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.082776 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:29.082762946 +0000 UTC m=+52.791057339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : configmap "kube-rbac-proxy" not found Sep 10 20:59:25.179435 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.179412 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zxhqt_33e80727-2e90-4091-84aa-90798cffb324/serve-healthcheck-canary/0.log" Sep 10 20:59:25.284432 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.284395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:25.284825 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.284535 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:59:25.284825 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.284593 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert podName:9c8ec021-c528-4678-be74-66f8da1edd93 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:27.284577271 +0000 UTC m=+50.992871635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vjq5j" (UID: "9c8ec021-c528-4678-be74-66f8da1edd93") : secret "networking-console-plugin-cert" not found Sep 10 20:59:25.463050 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.462974 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:25.463050 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.463022 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:25.463341 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:25.463328 2585 scope.go:117] "RemoveContainer" containerID="09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969" Sep 10 20:59:25.463488 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:25.463471 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-pf8tm_openshift-console-operator(2555a5e5-54fd-48e0-907a-4aef7afcb35c)\"" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" podUID="2555a5e5-54fd-48e0-907a-4aef7afcb35c" Sep 10 20:59:27.100162 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:27.100129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:27.100648 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:27.100293 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:59:27.100648 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:27.100373 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls podName:6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:43.100351945 +0000 UTC m=+66.808646331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls") pod "dns-default-t6h5z" (UID: "6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9") : secret "dns-default-metrics-tls" not found Sep 10 20:59:27.301700 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:27.301664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:27.301856 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:27.301798 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:59:27.301906 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:27.301858 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert podName:9c8ec021-c528-4678-be74-66f8da1edd93 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:31.301843474 +0000 UTC m=+55.010137838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vjq5j" (UID: "9c8ec021-c528-4678-be74-66f8da1edd93") : secret "networking-console-plugin-cert" not found Sep 10 20:59:29.116257 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:29.116225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:29.116653 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:29.116265 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:29.116653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:29.116372 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:29.116653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:29.116416 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:59:29.116653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:29.116436 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:37.116421231 +0000 UTC m=+60.824715629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:29.116653 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:29.116468 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:37.116456243 +0000 UTC m=+60.824750638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : configmap "kube-rbac-proxy" not found Sep 10 20:59:31.032117 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:31.032084 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:31.034424 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:31.034404 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnzzs\" (UID: \"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:31.077418 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:31.077396 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" Sep 10 20:59:31.334303 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:31.334242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:31.334428 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:31.334385 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:59:31.334467 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:31.334447 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert podName:9c8ec021-c528-4678-be74-66f8da1edd93 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.33443207 +0000 UTC m=+63.042726459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-vjq5j" (UID: "9c8ec021-c528-4678-be74-66f8da1edd93") : secret "networking-console-plugin-cert" not found Sep 10 20:59:31.353726 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:31.353700 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs"] Sep 10 20:59:32.192552 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:32.192510 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" event={"ID":"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de","Type":"ContainerStarted","Data":"3d19ccd26bd4461dc366a775361fca10d6333d5536be18aeb3bc07b40833e834"} Sep 10 20:59:33.197665 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:33.197571 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" event={"ID":"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de","Type":"ContainerStarted","Data":"f9278a03d4bb5735fbe9e883cb8fb48ad6afc3d955492c5f4b712abf705b873f"} Sep 10 20:59:33.197665 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:33.197643 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" event={"ID":"5c0e3aaf-8a76-4ea8-9d2d-3dc5af41f1de","Type":"ContainerStarted","Data":"39cd03434068d5cca1b461a7fb717102e9ae66cade4198e7ffe772ec792a37db"} Sep 10 20:59:33.221244 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:33.221202 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnzzs" podStartSLOduration=16.702759283 podStartE2EDuration="18.221187871s" totalImagesPullingTime="1.518428588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:33.219722892 +0000 UTC m=+56.928017290" watchObservedRunningTime="2026-09-10 20:59:33.221187871 +0000 UTC m=+56.929482259" Sep 10 20:59:36.184411 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:36.184379 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68vl" Sep 10 20:59:37.177568 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:37.177530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:37.177750 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:37.177662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:37.177796 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:37.177762 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:59:37.177830 ip-10-0-130-192 kubenswrapper[2585]: E0910 20:59:37.177817 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls podName:6abe2e34-0083-40d6-bd26-12d10effbbe0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:53.177801726 +0000 UTC m=+76.886096092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbf6p" (UID: "6abe2e34-0083-40d6-bd26-12d10effbbe0") : secret "insights-runtime-extractor-tls" not found Sep 10 20:59:37.178058 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:37.178037 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6abe2e34-0083-40d6-bd26-12d10effbbe0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:38.812049 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:38.812020 2585 scope.go:117] "RemoveContainer" containerID="09556b354797ae4970680a5c08e272f1a4c5e4c75f3b1cd4a426364b8a785969" Sep 10 20:59:39.215653 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.215552 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 20:59:39.215653 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.215640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" event={"ID":"2555a5e5-54fd-48e0-907a-4aef7afcb35c","Type":"ContainerStarted","Data":"b552e99dc14d7c97bff78f76cddff81c860310b9146a432476935c106f1b0771"} Sep 10 20:59:39.215935 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.215917 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:39.242921 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.242871 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" podStartSLOduration=20.307387746 podStartE2EDuration="24.242853706s" totalImagesPullingTime="3.93546596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:39.24225652 +0000 UTC m=+62.950550907" watchObservedRunningTime="2026-09-10 20:59:39.242853706 +0000 UTC m=+62.951148093" Sep 10 20:59:39.395751 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.395716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:39.398042 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.398019 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ec021-c528-4678-be74-66f8da1edd93-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-vjq5j\" (UID: \"9c8ec021-c528-4678-be74-66f8da1edd93\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:39.497200 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.497128 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lr75n\"" Sep 10 20:59:39.505288 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.505270 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" Sep 10 20:59:39.512069 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.512047 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-pf8tm" Sep 10 20:59:39.818201 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:39.818171 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j"] Sep 10 20:59:39.821341 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:39.821310 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ec021_c528_4678_be74_66f8da1edd93.slice/crio-fa31ffc6d53f02327aeabb0480597819347d7c538b4a78b6b9284c36769fae7b WatchSource:0}: Error finding container fa31ffc6d53f02327aeabb0480597819347d7c538b4a78b6b9284c36769fae7b: Status 404 returned error can't find the container with id fa31ffc6d53f02327aeabb0480597819347d7c538b4a78b6b9284c36769fae7b Sep 10 20:59:40.220153 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:40.220069 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" event={"ID":"9c8ec021-c528-4678-be74-66f8da1edd93","Type":"ContainerStarted","Data":"fa31ffc6d53f02327aeabb0480597819347d7c538b4a78b6b9284c36769fae7b"} Sep 10 20:59:41.224968 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:41.224871 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" event={"ID":"9c8ec021-c528-4678-be74-66f8da1edd93","Type":"ContainerStarted","Data":"78df659e06dbe86e1e48d79b47071772027f340302232f7806b8bf15c194ab8e"} Sep 10 20:59:41.254727 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:41.254674 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-vjq5j" podStartSLOduration=17.20794792 podStartE2EDuration="18.254660653s" totalImagesPullingTime="1.046712733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:41.254005007 +0000 UTC m=+64.962299393" watchObservedRunningTime="2026-09-10 20:59:41.254660653 +0000 UTC m=+64.962955039" Sep 10 20:59:42.518658 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.518630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:42.521867 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.521850 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 20:59:42.531297 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.531268 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0-original-pull-secret\") pod \"global-pull-secret-syncer-trv4s\" (UID: \"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0\") " pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:42.619717 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.619694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:42.619815 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.619748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:42.625796 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.625775 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 20:59:42.625846 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.625775 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 20:59:42.632016 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.631998 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85647140-f29b-4aac-ba00-20c319867dcd-metrics-certs\") pod \"network-metrics-daemon-fkjxm\" (UID: \"85647140-f29b-4aac-ba00-20c319867dcd\") " pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:42.632560 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.632535 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 20:59:42.642509 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.642490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vxs\" (UniqueName: \"kubernetes.io/projected/94843d1c-3563-4551-9ddd-a71aebaa186c-kube-api-access-t4vxs\") pod \"network-check-target-wlqrb\" (UID: \"94843d1c-3563-4551-9ddd-a71aebaa186c\") " pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:42.824764 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.824740 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-947bp\"" Sep 10 20:59:42.826848 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.826825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trv4s" Sep 10 20:59:42.831551 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.831534 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:42.833916 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.833895 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qc4z\"" Sep 10 20:59:42.842055 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:42.842032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fkjxm" Sep 10 20:59:43.114739 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.114669 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trv4s"] Sep 10 20:59:43.119077 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:43.119051 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6cb7b1_8cc9_438e_b98f_d5b70e2935f0.slice/crio-4f51aaed0eb1421c5ca006fce50fb3c38c4c51bad773febde7937133b8bcb6f9 WatchSource:0}: Error finding container 4f51aaed0eb1421c5ca006fce50fb3c38c4c51bad773febde7937133b8bcb6f9: Status 404 returned error can't find the container with id 4f51aaed0eb1421c5ca006fce50fb3c38c4c51bad773febde7937133b8bcb6f9 Sep 10 20:59:43.124046 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.124025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:43.124189 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.124168 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlqrb"] Sep 10 20:59:43.126688 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.126665 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9-metrics-tls\") pod \"dns-default-t6h5z\" (UID: \"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9\") " pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:43.127075 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:43.127057 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94843d1c_3563_4551_9ddd_a71aebaa186c.slice/crio-849edbd66fd86a096acd3ee6e93be78d0698135af90e18bd88e9f8d854d75d5d WatchSource:0}: Error finding container 849edbd66fd86a096acd3ee6e93be78d0698135af90e18bd88e9f8d854d75d5d: Status 404 returned error can't find the container with id 849edbd66fd86a096acd3ee6e93be78d0698135af90e18bd88e9f8d854d75d5d Sep 10 20:59:43.139106 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.139085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fkjxm"] Sep 10 20:59:43.141590 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:43.141567 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85647140_f29b_4aac_ba00_20c319867dcd.slice/crio-ab48ab8f239b1c7363c12c10e4ea4aaf0fc4efa8bc2931f70de6b262f003f770 WatchSource:0}: Error finding container ab48ab8f239b1c7363c12c10e4ea4aaf0fc4efa8bc2931f70de6b262f003f770: Status 404 returned error can't find the container with id ab48ab8f239b1c7363c12c10e4ea4aaf0fc4efa8bc2931f70de6b262f003f770 Sep 10 20:59:43.231969 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.231937 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlqrb" event={"ID":"94843d1c-3563-4551-9ddd-a71aebaa186c","Type":"ContainerStarted","Data":"849edbd66fd86a096acd3ee6e93be78d0698135af90e18bd88e9f8d854d75d5d"} Sep 10 20:59:43.233059 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.233025 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-trv4s" event={"ID":"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0","Type":"ContainerStarted","Data":"4f51aaed0eb1421c5ca006fce50fb3c38c4c51bad773febde7937133b8bcb6f9"} Sep 10 20:59:43.234125 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.234091 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkjxm" event={"ID":"85647140-f29b-4aac-ba00-20c319867dcd","Type":"ContainerStarted","Data":"ab48ab8f239b1c7363c12c10e4ea4aaf0fc4efa8bc2931f70de6b262f003f770"} Sep 10 20:59:43.368249 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.368176 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vl2hm\"" Sep 10 20:59:43.375992 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.375969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:43.687883 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:43.687806 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t6h5z"] Sep 10 20:59:43.692588 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:43.692527 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2cf1ee_cd72_44b5_b63a_3fb8ef00aae9.slice/crio-edacfd6f17026763be0d5fc193ee6e5dfff3421304629bf3773f8e54baab5f1c WatchSource:0}: Error finding container edacfd6f17026763be0d5fc193ee6e5dfff3421304629bf3773f8e54baab5f1c: Status 404 returned error can't find the container with id edacfd6f17026763be0d5fc193ee6e5dfff3421304629bf3773f8e54baab5f1c Sep 10 20:59:44.240444 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:44.240409 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6h5z" event={"ID":"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9","Type":"ContainerStarted","Data":"edacfd6f17026763be0d5fc193ee6e5dfff3421304629bf3773f8e54baab5f1c"} Sep 10 20:59:45.246889 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:45.246811 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkjxm" event={"ID":"85647140-f29b-4aac-ba00-20c319867dcd","Type":"ContainerStarted","Data":"90074236ff326c6342e70ec5dbe91c70cdf0679d6a2fce0b5c075d71cdac12ad"} Sep 10 20:59:45.246889 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:45.246872 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fkjxm" event={"ID":"85647140-f29b-4aac-ba00-20c319867dcd","Type":"ContainerStarted","Data":"8620c27812e4347b7e41e037b5552ef046de3558b26739a40775fa820eb05990"} Sep 10 20:59:45.277526 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:45.276398 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fkjxm" podStartSLOduration=66.732065626 podStartE2EDuration="1m8.276377805s" totalImagesPullingTime="1.544312179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:45.27589505 +0000 UTC m=+68.984189444" watchObservedRunningTime="2026-09-10 20:59:45.276377805 +0000 UTC m=+68.984672191" Sep 10 20:59:49.262622 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.262581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlqrb" event={"ID":"94843d1c-3563-4551-9ddd-a71aebaa186c","Type":"ContainerStarted","Data":"77687d1b8ab4c093bdf586522767ada8316bacc895699304f167894fc9da9431"} Sep 10 20:59:49.263173 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.262694 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 20:59:49.264399 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.264374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6h5z" event={"ID":"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9","Type":"ContainerStarted","Data":"a53051569519f2260097b3b482a77886a819980e80be0e6bc4bec8637f1d67ce"} Sep 10 20:59:49.264515 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.264421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t6h5z" event={"ID":"6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9","Type":"ContainerStarted","Data":"3d1d1ee4b37bfa07cd8b780ed4794d387622eb235ddab310003ada0363cbd6c7"} Sep 10 20:59:49.264515 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.264446 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:49.265777 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.265753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-trv4s" event={"ID":"0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0","Type":"ContainerStarted","Data":"12d230dced6bec919f0458dc8038b95319dc90df66ef01d0bb089cfbb1f2338f"} Sep 10 20:59:49.318587 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.318542 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wlqrb" podStartSLOduration=67.198823904 podStartE2EDuration="1m12.31852674s" totalImagesPullingTime="5.119702836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:49.295836742 +0000 UTC m=+73.004131128" watchObservedRunningTime="2026-09-10 20:59:49.31852674 +0000 UTC m=+73.026821185" Sep 10 20:59:49.318820 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.318800 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-trv4s" podStartSLOduration=67.174569186 podStartE2EDuration="1m12.318791375s" totalImagesPullingTime="5.144222189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:49.317760854 +0000 UTC m=+73.026055241" watchObservedRunningTime="2026-09-10 20:59:49.318791375 +0000 UTC m=+73.027085761" Sep 10 20:59:49.359009 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:49.358959 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t6h5z" podStartSLOduration=33.803347787 podStartE2EDuration="38.358948055s" totalImagesPullingTime="4.555600268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:49.357889836 +0000 UTC m=+73.066184222" watchObservedRunningTime="2026-09-10 20:59:49.358948055 +0000 UTC m=+73.067242442" Sep 10 20:59:52.392993 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.392961 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt"] Sep 10 20:59:52.423776 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.423750 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd"] Sep 10 20:59:52.423917 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.423879 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.444515 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.444490 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 20:59:52.447654 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.447636 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 20:59:52.447767 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.447640 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.453431 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.453411 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-697wn\"" Sep 10 20:59:52.453790 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.453765 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 20:59:52.456412 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.456382 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd"] Sep 10 20:59:52.457820 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.457787 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt"] Sep 10 20:59:52.458672 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.458651 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 20:59:52.493624 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.493592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnqm\" (UniqueName: \"kubernetes.io/projected/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-kube-api-access-6vnqm\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.493737 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.493718 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-klusterlet-config\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.493780 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.493755 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-tmp\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.520555 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.520533 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 20:59:52.594626 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.594583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f27b59aa-376f-415c-9bb0-a0f3a24314d0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.594764 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.594647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-klusterlet-config\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.594764 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.594674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrs4\" (UniqueName: \"kubernetes.io/projected/f27b59aa-376f-415c-9bb0-a0f3a24314d0-kube-api-access-skrs4\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.594764 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.594699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-tmp\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.594764 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.594723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnqm\" (UniqueName: \"kubernetes.io/projected/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-kube-api-access-6vnqm\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.595090 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.595071 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-tmp\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.597204 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.597183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-klusterlet-config\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.626737 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.626714 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnqm\" (UniqueName: \"kubernetes.io/projected/e2c16cae-1d03-45b7-be15-5dd6a1d6d129-kube-api-access-6vnqm\") pod \"klusterlet-addon-workmgr-8659f9df4-nnrdt\" (UID: \"e2c16cae-1d03-45b7-be15-5dd6a1d6d129\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.695942 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.695862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f27b59aa-376f-415c-9bb0-a0f3a24314d0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.695942 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.695902 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrs4\" (UniqueName: \"kubernetes.io/projected/f27b59aa-376f-415c-9bb0-a0f3a24314d0-kube-api-access-skrs4\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.698606 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.698574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f27b59aa-376f-415c-9bb0-a0f3a24314d0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.733497 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.733470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:52.752120 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.752091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrs4\" (UniqueName: \"kubernetes.io/projected/f27b59aa-376f-415c-9bb0-a0f3a24314d0-kube-api-access-skrs4\") pod \"managed-serviceaccount-addon-agent-5c9bd976b8-glccd\" (UID: \"f27b59aa-376f-415c-9bb0-a0f3a24314d0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:52.764685 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:52.764667 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" Sep 10 20:59:53.108803 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.108715 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd"] Sep 10 20:59:53.111377 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:53.111351 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf27b59aa_376f_415c_9bb0_a0f3a24314d0.slice/crio-4c35b7b533d5f973dfb6f58c131a077a12f32ddc5463940021d5c4d2984e79f2 WatchSource:0}: Error finding container 4c35b7b533d5f973dfb6f58c131a077a12f32ddc5463940021d5c4d2984e79f2: Status 404 returned error can't find the container with id 4c35b7b533d5f973dfb6f58c131a077a12f32ddc5463940021d5c4d2984e79f2 Sep 10 20:59:53.132997 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.132978 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt"] Sep 10 20:59:53.134798 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:53.134767 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c16cae_1d03_45b7_be15_5dd6a1d6d129.slice/crio-bfa58f3ca218b221419a93a8c380427ae7f7daf2ed1147bea10a74b0a7c04347 WatchSource:0}: Error finding container bfa58f3ca218b221419a93a8c380427ae7f7daf2ed1147bea10a74b0a7c04347: Status 404 returned error can't find the container with id bfa58f3ca218b221419a93a8c380427ae7f7daf2ed1147bea10a74b0a7c04347 Sep 10 20:59:53.200117 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.200097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:53.202146 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.202129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6abe2e34-0083-40d6-bd26-12d10effbbe0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbf6p\" (UID: \"6abe2e34-0083-40d6-bd26-12d10effbbe0\") " pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:53.280067 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.280033 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" event={"ID":"e2c16cae-1d03-45b7-be15-5dd6a1d6d129","Type":"ContainerStarted","Data":"bfa58f3ca218b221419a93a8c380427ae7f7daf2ed1147bea10a74b0a7c04347"} Sep 10 20:59:53.281159 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.281138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" event={"ID":"f27b59aa-376f-415c-9bb0-a0f3a24314d0","Type":"ContainerStarted","Data":"4c35b7b533d5f973dfb6f58c131a077a12f32ddc5463940021d5c4d2984e79f2"} Sep 10 20:59:53.380597 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.380545 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7vxwx\"" Sep 10 20:59:53.387391 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.387376 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbf6p" Sep 10 20:59:53.714647 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:53.714549 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbf6p"] Sep 10 20:59:53.717988 ip-10-0-130-192 kubenswrapper[2585]: W0910 20:59:53.717955 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe2e34_0083_40d6_bd26_12d10effbbe0.slice/crio-1c2180fee767a41f746b7c3a25c2d8904f50c627e68eba45d0c9296c4d4bcd5e WatchSource:0}: Error finding container 1c2180fee767a41f746b7c3a25c2d8904f50c627e68eba45d0c9296c4d4bcd5e: Status 404 returned error can't find the container with id 1c2180fee767a41f746b7c3a25c2d8904f50c627e68eba45d0c9296c4d4bcd5e Sep 10 20:59:54.292076 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:54.292036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbf6p" event={"ID":"6abe2e34-0083-40d6-bd26-12d10effbbe0","Type":"ContainerStarted","Data":"2e093d9332cf841b86af988c13466c350bdd97f2826a51fd1b4eaccede15e799"} Sep 10 20:59:54.292265 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:54.292116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbf6p" event={"ID":"6abe2e34-0083-40d6-bd26-12d10effbbe0","Type":"ContainerStarted","Data":"1c2180fee767a41f746b7c3a25c2d8904f50c627e68eba45d0c9296c4d4bcd5e"} Sep 10 20:59:55.299199 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:55.299159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbf6p" event={"ID":"6abe2e34-0083-40d6-bd26-12d10effbbe0","Type":"ContainerStarted","Data":"bf7d6e6ce109805d3dfb5ea041fc34cdc5a82ea00cce1ad170f0676d564934e8"} Sep 10 20:59:59.270826 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.270707 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t6h5z" Sep 10 20:59:59.321456 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.321421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" event={"ID":"e2c16cae-1d03-45b7-be15-5dd6a1d6d129","Type":"ContainerStarted","Data":"71449deba82150ee80d7f84560b6a9f6f6b216784c5f2dbda915b15b9599606e"} Sep 10 20:59:59.321653 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.321513 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:59.323284 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.323261 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" Sep 10 20:59:59.323422 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.323304 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" event={"ID":"f27b59aa-376f-415c-9bb0-a0f3a24314d0","Type":"ContainerStarted","Data":"490b94d30d793a368b641a3d729f8f5a8f875fd87cca4a201953cba44f884e21"} Sep 10 20:59:59.325542 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.325497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbf6p" event={"ID":"6abe2e34-0083-40d6-bd26-12d10effbbe0","Type":"ContainerStarted","Data":"40c504ec6a618f58dda5bc96b83b5d56cc1654076a8250078bb960681daa2efe"} Sep 10 20:59:59.380236 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.380187 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8659f9df4-nnrdt" podStartSLOduration=2.258054055 podStartE2EDuration="7.380172933s" totalImagesPullingTime="5.122118878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:59.379993089 +0000 UTC m=+83.088287475" watchObservedRunningTime="2026-09-10 20:59:59.380172933 +0000 UTC m=+83.088467316" Sep 10 20:59:59.409702 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.409653 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kbf6p" podStartSLOduration=34.088029539 podStartE2EDuration="38.4096401s" totalImagesPullingTime="4.321610561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:59.409432821 +0000 UTC m=+83.117727208" watchObservedRunningTime="2026-09-10 20:59:59.4096401 +0000 UTC m=+83.117934486" Sep 10 20:59:59.479370 ip-10-0-130-192 kubenswrapper[2585]: I0910 20:59:59.479314 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9bd976b8-glccd" podStartSLOduration=2.334558686 podStartE2EDuration="7.47929706s" totalImagesPullingTime="5.144738374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:59.478141397 +0000 UTC m=+83.186435782" watchObservedRunningTime="2026-09-10 20:59:59.47929706 +0000 UTC m=+83.187591444" Sep 10 21:00:02.051382 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.051350 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:02.055500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.055477 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.059701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.059679 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 21:00:02.060078 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060063 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 21:00:02.060120 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060088 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 21:00:02.060362 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 21:00:02.060418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060398 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dtlm8\"" Sep 10 21:00:02.060418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060409 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 21:00:02.060513 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060434 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 21:00:02.060513 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.060438 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 21:00:02.077151 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.077126 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:02.166922 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.166894 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.167086 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.166926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.167086 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.166945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.167086 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.166971 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.167236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.167077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.167236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.167127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hl4\" (UniqueName: \"kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hl4\" (UniqueName: \"kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.267945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.267918 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.268465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.268442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.268587 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.268569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.268587 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.268575 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.270382 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.270358 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.270451 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.270443 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.286771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.286744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hl4\" (UniqueName: \"kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4\") pod \"console-755cfbd8df-pfbb5\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.364712 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.364652 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:02.537256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.537226 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-666sp"] Sep 10 21:00:02.540633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.540603 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.546278 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.546254 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 21:00:02.546645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.546629 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 21:00:02.547627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.547434 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m26fb\"" Sep 10 21:00:02.547627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.547444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 21:00:02.547627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.547508 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 21:00:02.547627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.547446 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 21:00:02.547627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.547514 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 21:00:02.657776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.657717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:02.660977 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:02.660954 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7101912a_127c_4397_94bb_8dc604e2ca96.slice/crio-a116c55df3f0fe1feb6a8da21ff8e5b8b04c0a69238d64ffc444724a7e47372b WatchSource:0}: Error finding container a116c55df3f0fe1feb6a8da21ff8e5b8b04c0a69238d64ffc444724a7e47372b: Status 404 returned error can't find the container with id a116c55df3f0fe1feb6a8da21ff8e5b8b04c0a69238d64ffc444724a7e47372b Sep 10 21:00:02.670699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670680 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-metrics-client-ca\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.670774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.670774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-textfile\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.670774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.670925 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670867 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-root\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.670925 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-tls\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.671023 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-sys\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.671023 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.670985 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7p6\" (UniqueName: \"kubernetes.io/projected/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-kube-api-access-5d7p6\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.671106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.671019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-wtmp\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772093 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-textfile\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-root\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-tls\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772177 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-sys\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7p6\" (UniqueName: \"kubernetes.io/projected/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-kube-api-access-5d7p6\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772240 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-wtmp\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772244 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-sys\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-root\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-metrics-client-ca\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772322 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-wtmp\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-textfile\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772775 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772745 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-accelerators-collector-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.772817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.772800 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-metrics-client-ca\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.774438 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.774419 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-tls\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.774565 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.774547 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.787672 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.787649 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7p6\" (UniqueName: \"kubernetes.io/projected/7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d-kube-api-access-5d7p6\") pod \"node-exporter-666sp\" (UID: \"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d\") " pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.849489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:02.849466 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666sp" Sep 10 21:00:02.860647 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:02.860602 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0132b3_2f37_4f0b_9a87_b9b51bcbfa9d.slice/crio-73d85156176dd01348abbd8c57ae2554ec93f7e5f3c845eff38b08f7ca821ffc WatchSource:0}: Error finding container 73d85156176dd01348abbd8c57ae2554ec93f7e5f3c845eff38b08f7ca821ffc: Status 404 returned error can't find the container with id 73d85156176dd01348abbd8c57ae2554ec93f7e5f3c845eff38b08f7ca821ffc Sep 10 21:00:03.340463 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.340431 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755cfbd8df-pfbb5" event={"ID":"7101912a-127c-4397-94bb-8dc604e2ca96","Type":"ContainerStarted","Data":"a116c55df3f0fe1feb6a8da21ff8e5b8b04c0a69238d64ffc444724a7e47372b"} Sep 10 21:00:03.341582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.341555 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666sp" event={"ID":"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d","Type":"ContainerStarted","Data":"73d85156176dd01348abbd8c57ae2554ec93f7e5f3c845eff38b08f7ca821ffc"} Sep 10 21:00:03.492206 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.492174 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:00:03.496907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.496889 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.506196 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506155 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 21:00:03.506363 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506215 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 21:00:03.506363 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506232 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 21:00:03.506500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506378 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 21:00:03.506500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506460 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 21:00:03.506605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.506508 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zc2w7\"" Sep 10 21:00:03.507071 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.507052 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 21:00:03.507150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.507133 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 21:00:03.507371 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.507357 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 21:00:03.508060 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.508045 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 21:00:03.532323 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.532295 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:00:03.577298 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577267 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577331 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zs9d\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577425 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577454 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.577928 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.577757 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.678762 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.678762 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.678762 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.678762 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678800 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678839 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zs9d\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:00:03.678879 2585 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:00:03.678914 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle podName:792683bb-1b80-401c-9bab-63746ab78e2c nodeName:}" failed. No retries permitted until 2026-09-10 21:00:04.178892099 +0000 UTC m=+87.887186469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c") : configmap references non-existent config key: ca-bundle.crt Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.678972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.679003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.679038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679543 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.679479 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.679543 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:00:03.679535 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls podName:792683bb-1b80-401c-9bab-63746ab78e2c nodeName:}" failed. No retries permitted until 2026-09-10 21:00:04.179515847 +0000 UTC m=+87.887810227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c") : secret "alertmanager-main-tls" not found Sep 10 21:00:03.679685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.679549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.681702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.681670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.681979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.681913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682164 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682356 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682334 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682406 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682335 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.682843 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.682827 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:03.693255 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:03.693233 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zs9d\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.183864 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.183827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.183864 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.183869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.184786 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.184756 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.186669 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.186646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.246436 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.246404 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:04.249661 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.249644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.264911 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.264889 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 21:00:04.280756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.280736 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:04.385795 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.385763 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.385806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.385829 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.385890 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfjm\" (UniqueName: \"kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.385929 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.386049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.386247 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.386111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.406065 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.406029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:00:04.487549 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487470 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487549 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487672 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.487778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.487729 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfjm\" (UniqueName: \"kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.488705 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.488265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.488705 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.488298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.489189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.488993 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.489189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.489002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.491163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.491143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.491497 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.491480 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.500544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.500519 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfjm\" (UniqueName: \"kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm\") pod \"console-54f8cdf484-l66t6\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.561209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.561024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:04.862295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.862265 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:00:04.865639 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:04.865198 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792683bb_1b80_401c_9bab_63746ab78e2c.slice/crio-085aa7d4cb60fea25dc350d24d77f91afe22e1d686f9c992651586bcbe8d1612 WatchSource:0}: Error finding container 085aa7d4cb60fea25dc350d24d77f91afe22e1d686f9c992651586bcbe8d1612: Status 404 returned error can't find the container with id 085aa7d4cb60fea25dc350d24d77f91afe22e1d686f9c992651586bcbe8d1612 Sep 10 21:00:04.908243 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:04.908131 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:04.910480 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:04.910454 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8469c6f_7243_4c2d_a3bc_861f54a640e7.slice/crio-5e3eefcbe97783e7f48f66736ad90d067a482fc7c37fb507a445ee46aeaeb9b8 WatchSource:0}: Error finding container 5e3eefcbe97783e7f48f66736ad90d067a482fc7c37fb507a445ee46aeaeb9b8: Status 404 returned error can't find the container with id 5e3eefcbe97783e7f48f66736ad90d067a482fc7c37fb507a445ee46aeaeb9b8 Sep 10 21:00:05.349803 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.349765 2585 generic.go:356] "Generic (PLEG): container finished" podID="7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d" containerID="c19fc4537b5d1e507a1d671389db44ac8f8cf74aa734f376b8029d0f88e44113" exitCode=0 Sep 10 21:00:05.349975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.349850 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666sp" event={"ID":"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d","Type":"ContainerDied","Data":"c19fc4537b5d1e507a1d671389db44ac8f8cf74aa734f376b8029d0f88e44113"} Sep 10 21:00:05.351329 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.351298 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8cdf484-l66t6" event={"ID":"f8469c6f-7243-4c2d-a3bc-861f54a640e7","Type":"ContainerStarted","Data":"5e3eefcbe97783e7f48f66736ad90d067a482fc7c37fb507a445ee46aeaeb9b8"} Sep 10 21:00:05.352664 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.352637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"085aa7d4cb60fea25dc350d24d77f91afe22e1d686f9c992651586bcbe8d1612"} Sep 10 21:00:05.507138 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.506827 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5565f8586f-dgv59"] Sep 10 21:00:05.511101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.511081 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.517703 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.517648 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 21:00:05.518669 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.518199 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cl3t01dpr6ng8\"" Sep 10 21:00:05.518669 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.518463 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 21:00:05.521172 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.520725 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 21:00:05.521172 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.520966 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-crqmh\"" Sep 10 21:00:05.521321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.521209 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 21:00:05.521480 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.521411 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 21:00:05.548994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.548971 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5565f8586f-dgv59"] Sep 10 21:00:05.599226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2sh\" (UniqueName: \"kubernetes.io/projected/42aa9918-064e-42c8-951f-7e3cfa20ad97-kube-api-access-xc2sh\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-grpc-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599310 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aa9918-064e-42c8-951f-7e3cfa20ad97-metrics-client-ca\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599430 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.599652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.599566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2sh\" (UniqueName: \"kubernetes.io/projected/42aa9918-064e-42c8-951f-7e3cfa20ad97-kube-api-access-xc2sh\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-grpc-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.700994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.700979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aa9918-064e-42c8-951f-7e3cfa20ad97-metrics-client-ca\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.701211 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.701020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.702126 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.702093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aa9918-064e-42c8-951f-7e3cfa20ad97-metrics-client-ca\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.704194 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.704167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.704686 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.704579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.705202 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.705145 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.705202 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.705152 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.705357 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.705288 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-grpc-tls\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.706429 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.706390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aa9918-064e-42c8-951f-7e3cfa20ad97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.721207 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.721181 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2sh\" (UniqueName: \"kubernetes.io/projected/42aa9918-064e-42c8-951f-7e3cfa20ad97-kube-api-access-xc2sh\") pod \"thanos-querier-5565f8586f-dgv59\" (UID: \"42aa9918-064e-42c8-951f-7e3cfa20ad97\") " pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:05.824322 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:05.824284 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:06.359371 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:06.359330 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666sp" event={"ID":"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d","Type":"ContainerStarted","Data":"a09d7279a44a61375d9d1b047d187a5beace4eef9a1b062481a7da0f5af0ebfc"} Sep 10 21:00:06.359532 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:06.359388 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666sp" event={"ID":"7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d","Type":"ContainerStarted","Data":"06d663032be2f9e942dc8e72b81a760b524c09376541bd519e372a3a114c5ed6"} Sep 10 21:00:07.155184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.154760 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-666sp" podStartSLOduration=3.335982092 podStartE2EDuration="5.154740989s" totalImagesPullingTime="1.747736249s" totalInitContainerRuntime="71.022648ms" isStatefulPod=true podCreationTimestamp="2026-09-10 21:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:06.387810742 +0000 UTC m=+90.096105128" watchObservedRunningTime="2026-09-10 21:00:07.154740989 +0000 UTC m=+90.863035376" Sep 10 21:00:07.156267 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.156243 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr"] Sep 10 21:00:07.159673 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.159654 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.171313 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.171288 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:07.175137 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.175090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr"] Sep 10 21:00:07.178891 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.178862 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j5qvt\"" Sep 10 21:00:07.179005 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.178985 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 21:00:07.315346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.315272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e85a6da-560f-4e4c-b760-adc779d7f37a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.315346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.315314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vvz\" (UniqueName: \"kubernetes.io/projected/8e85a6da-560f-4e4c-b760-adc779d7f37a-kube-api-access-p6vvz\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.365094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.365064 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755cfbd8df-pfbb5" event={"ID":"7101912a-127c-4397-94bb-8dc604e2ca96","Type":"ContainerStarted","Data":"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889"} Sep 10 21:00:07.366858 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.366830 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf" exitCode=0 Sep 10 21:00:07.366990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.366914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf"} Sep 10 21:00:07.368704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.368679 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8cdf484-l66t6" event={"ID":"f8469c6f-7243-4c2d-a3bc-861f54a640e7","Type":"ContainerStarted","Data":"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911"} Sep 10 21:00:07.390795 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.390774 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5565f8586f-dgv59"] Sep 10 21:00:07.392492 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:07.392461 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42aa9918_064e_42c8_951f_7e3cfa20ad97.slice/crio-414280b742bbd23772fab28cef1cbe70b299d6494e248d3c972839c22091c95a WatchSource:0}: Error finding container 414280b742bbd23772fab28cef1cbe70b299d6494e248d3c972839c22091c95a: Status 404 returned error can't find the container with id 414280b742bbd23772fab28cef1cbe70b299d6494e248d3c972839c22091c95a Sep 10 21:00:07.407381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.407339 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755cfbd8df-pfbb5" podStartSLOduration=1.025187661 podStartE2EDuration="5.407323702s" totalImagesPullingTime="4.382136041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:07.406944137 +0000 UTC m=+91.115238523" watchObservedRunningTime="2026-09-10 21:00:07.407323702 +0000 UTC m=+91.115618089" Sep 10 21:00:07.416439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.416416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vvz\" (UniqueName: \"kubernetes.io/projected/8e85a6da-560f-4e4c-b760-adc779d7f37a-kube-api-access-p6vvz\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.416512 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.416487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e85a6da-560f-4e4c-b760-adc779d7f37a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.418814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.418795 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e85a6da-560f-4e4c-b760-adc779d7f37a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.427307 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.427286 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vvz\" (UniqueName: \"kubernetes.io/projected/8e85a6da-560f-4e4c-b760-adc779d7f37a-kube-api-access-p6vvz\") pod \"monitoring-plugin-5d85778b9d-2l2fr\" (UID: \"8e85a6da-560f-4e4c-b760-adc779d7f37a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.435057 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.435017 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54f8cdf484-l66t6" podStartSLOduration=1.301004895 podStartE2EDuration="3.435003555s" totalImagesPullingTime="2.13399866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:07.433364633 +0000 UTC m=+91.141659019" watchObservedRunningTime="2026-09-10 21:00:07.435003555 +0000 UTC m=+91.143297941" Sep 10 21:00:07.514604 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.514546 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:07.793968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:07.793944 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr"] Sep 10 21:00:07.796256 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:07.796229 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e85a6da_560f_4e4c_b760_adc779d7f37a.slice/crio-de4126e758daac4da2d3161b5b27d5e37541e6a5c34c91cce270d7c14224cf63 WatchSource:0}: Error finding container de4126e758daac4da2d3161b5b27d5e37541e6a5c34c91cce270d7c14224cf63: Status 404 returned error can't find the container with id de4126e758daac4da2d3161b5b27d5e37541e6a5c34c91cce270d7c14224cf63 Sep 10 21:00:08.378098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:08.377984 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" event={"ID":"8e85a6da-560f-4e4c-b760-adc779d7f37a","Type":"ContainerStarted","Data":"de4126e758daac4da2d3161b5b27d5e37541e6a5c34c91cce270d7c14224cf63"} Sep 10 21:00:08.382645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:08.382518 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"414280b742bbd23772fab28cef1cbe70b299d6494e248d3c972839c22091c95a"} Sep 10 21:00:10.394368 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.394291 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65"} Sep 10 21:00:10.394742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.394388 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a"} Sep 10 21:00:10.394742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.394416 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959"} Sep 10 21:00:10.394742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.394448 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69"} Sep 10 21:00:10.394742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.394473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f"} Sep 10 21:00:10.395765 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.395733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" event={"ID":"8e85a6da-560f-4e4c-b760-adc779d7f37a","Type":"ContainerStarted","Data":"4fd5e8742a679cacbbc7250e087d36905ea28a9d7a4f268f91c1936fe7ef9cee"} Sep 10 21:00:10.395901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.395866 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:10.398060 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.398033 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"c2c0b5e1380037e6a4e3d118116351571b29554b47dcc94a839e52fb62bd7aa2"} Sep 10 21:00:10.398155 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.398090 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"89d76eefa3c660e2f8854fe3f4a1904c51bb2de762a7ce537f6b02893b782b7c"} Sep 10 21:00:10.398155 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.398118 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"55e065199f3a65e33b2d6ea1b861430b797f3953027cfb86044d220e5bb614a2"} Sep 10 21:00:10.401085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.401067 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" Sep 10 21:00:10.419581 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:10.419537 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2l2fr" podStartSLOduration=1.335741217 podStartE2EDuration="3.419524622s" totalImagesPullingTime="2.083783405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:10.417924078 +0000 UTC m=+94.126218465" watchObservedRunningTime="2026-09-10 21:00:10.419524622 +0000 UTC m=+94.127819007" Sep 10 21:00:11.404908 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.404871 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"14e3bbfb761bec12c10e5f72cb1349d0ea397b3da1cb40dc20e396132cb29362"} Sep 10 21:00:11.405352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.404970 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:11.405352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.405017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"f434210569c37a7fd0885dcda494a8bc5119c8b0350320f490adaebab0fbe6af"} Sep 10 21:00:11.405352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.405067 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" event={"ID":"42aa9918-064e-42c8-951f-7e3cfa20ad97","Type":"ContainerStarted","Data":"c4f725f69a905a8dec01e7f754fb6de97b60714a77aa8f0898a4bf9bdd97d462"} Sep 10 21:00:11.408165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.408139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerStarted","Data":"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2"} Sep 10 21:00:11.444500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.444456 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" podStartSLOduration=3.254270756 podStartE2EDuration="6.444443001s" totalImagesPullingTime="3.190172245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:11.444033793 +0000 UTC m=+95.152328179" watchObservedRunningTime="2026-09-10 21:00:11.444443001 +0000 UTC m=+95.152737451" Sep 10 21:00:11.492008 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:11.491963 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.213607106 podStartE2EDuration="8.491949131s" totalImagesPullingTime="5.256970669s" totalInitContainerRuntime="21.371356ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:11.491472579 +0000 UTC m=+95.199766965" watchObservedRunningTime="2026-09-10 21:00:11.491949131 +0000 UTC m=+95.200243572" Sep 10 21:00:12.364896 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:12.364869 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:14.561140 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:14.561098 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:14.561538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:14.561158 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:14.566150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:14.566127 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:15.430192 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:15.430167 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:16.475172 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.475141 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:00:16.480861 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.480840 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.529736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.529708 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:00:16.594339 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594303 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594339 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594529 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594360 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nppm\" (UniqueName: \"kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594529 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594476 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594529 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594640 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.594640 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.594592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696149 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696320 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nppm\" (UniqueName: \"kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696320 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696296 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696383 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696383 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696463 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.696975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.696944 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.697094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.697004 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.697094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.697043 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.697206 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.697189 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.698499 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.698481 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.698779 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.698757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.714081 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.714054 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nppm\" (UniqueName: \"kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm\") pod \"console-6f8c9b495-fwpmg\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:16.789303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:16.789281 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:17.091731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:17.091662 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:00:17.094408 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:00:17.094379 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818e5b32_0413_4c64_bb05_02649bded1b5.slice/crio-d9d4b9080a06b653f94937d03a28eedefc6837dbf7fae6fa5a9ee8a033ee2735 WatchSource:0}: Error finding container d9d4b9080a06b653f94937d03a28eedefc6837dbf7fae6fa5a9ee8a033ee2735: Status 404 returned error can't find the container with id d9d4b9080a06b653f94937d03a28eedefc6837dbf7fae6fa5a9ee8a033ee2735 Sep 10 21:00:17.418823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:17.418748 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5565f8586f-dgv59" Sep 10 21:00:17.436078 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:17.436040 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8c9b495-fwpmg" event={"ID":"818e5b32-0413-4c64-bb05-02649bded1b5","Type":"ContainerStarted","Data":"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac"} Sep 10 21:00:17.436219 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:17.436094 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8c9b495-fwpmg" event={"ID":"818e5b32-0413-4c64-bb05-02649bded1b5","Type":"ContainerStarted","Data":"d9d4b9080a06b653f94937d03a28eedefc6837dbf7fae6fa5a9ee8a033ee2735"} Sep 10 21:00:17.486999 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:17.486937 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f8c9b495-fwpmg" podStartSLOduration=1.486914654 podStartE2EDuration="1.486914654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:17.484797467 +0000 UTC m=+101.193091865" watchObservedRunningTime="2026-09-10 21:00:17.486914654 +0000 UTC m=+101.195209041" Sep 10 21:00:20.270935 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:20.270903 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wlqrb" Sep 10 21:00:26.472797 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.472762 2585 generic.go:356] "Generic (PLEG): container finished" podID="4c728a21-5a9c-49d4-8e05-dbdbc6ded43a" containerID="e45c3fd1d5ed99311523b34d301c9983dbd07072673816daf6014ad12e38af5d" exitCode=0 Sep 10 21:00:26.473176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.472839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x964d" event={"ID":"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a","Type":"ContainerDied","Data":"e45c3fd1d5ed99311523b34d301c9983dbd07072673816daf6014ad12e38af5d"} Sep 10 21:00:26.473216 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.473193 2585 scope.go:117] "RemoveContainer" containerID="e45c3fd1d5ed99311523b34d301c9983dbd07072673816daf6014ad12e38af5d" Sep 10 21:00:26.789428 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.789402 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:26.789699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.789678 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:26.794978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:26.794951 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:27.479148 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:27.479111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x964d" event={"ID":"4c728a21-5a9c-49d4-8e05-dbdbc6ded43a","Type":"ContainerStarted","Data":"507a4f437966a8564e8ce66e8bbfe9d214751652bd73f358aebb8e406ee517bb"} Sep 10 21:00:27.482887 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:27.482869 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:00:27.631796 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:27.631765 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:33.411430 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.411365 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-755cfbd8df-pfbb5" podUID="7101912a-127c-4397-94bb-8dc604e2ca96" containerName="console" containerID="cri-o://82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889" gracePeriod=15 Sep 10 21:00:33.697389 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.697361 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:33.735699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735672 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.735875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735744 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.735875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735770 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.735875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735798 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.735875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735827 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.736086 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.735875 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5hl4\" (UniqueName: \"kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4\") pod \"7101912a-127c-4397-94bb-8dc604e2ca96\" (UID: \"7101912a-127c-4397-94bb-8dc604e2ca96\") " Sep 10 21:00:33.736195 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.736170 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:33.736258 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.736226 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config" (OuterVolumeSpecName: "console-config") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:33.736320 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.736252 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca" (OuterVolumeSpecName: "service-ca") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:33.738178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.738152 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4" (OuterVolumeSpecName: "kube-api-access-t5hl4") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "kube-api-access-t5hl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:00:33.738282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.738180 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:00:33.738413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.738391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7101912a-127c-4397-94bb-8dc604e2ca96" (UID: "7101912a-127c-4397-94bb-8dc604e2ca96"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:00:33.837361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837325 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:33.837361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837354 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-oauth-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:33.837361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837368 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-console-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:33.837577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837392 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7101912a-127c-4397-94bb-8dc604e2ca96-console-oauth-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:33.837577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837405 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7101912a-127c-4397-94bb-8dc604e2ca96-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:33.837577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:33.837417 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hl4\" (UniqueName: \"kubernetes.io/projected/7101912a-127c-4397-94bb-8dc604e2ca96-kube-api-access-t5hl4\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:34.507782 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.507745 2585 generic.go:356] "Generic (PLEG): container finished" podID="7101912a-127c-4397-94bb-8dc604e2ca96" containerID="82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889" exitCode=0 Sep 10 21:00:34.508183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.507790 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755cfbd8df-pfbb5" Sep 10 21:00:34.508183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.507805 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755cfbd8df-pfbb5" event={"ID":"7101912a-127c-4397-94bb-8dc604e2ca96","Type":"ContainerDied","Data":"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889"} Sep 10 21:00:34.508183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.507844 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755cfbd8df-pfbb5" event={"ID":"7101912a-127c-4397-94bb-8dc604e2ca96","Type":"ContainerDied","Data":"a116c55df3f0fe1feb6a8da21ff8e5b8b04c0a69238d64ffc444724a7e47372b"} Sep 10 21:00:34.508183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.507864 2585 scope.go:117] "RemoveContainer" containerID="82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889" Sep 10 21:00:34.516778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.516753 2585 scope.go:117] "RemoveContainer" containerID="82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889" Sep 10 21:00:34.517045 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:00:34.517026 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889\": container with ID starting with 82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889 not found: ID does not exist" containerID="82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889" Sep 10 21:00:34.517095 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.517054 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889"} err="failed to get container status \"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889\": rpc error: code = NotFound desc = could not find container \"82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889\": container with ID starting with 82e71d7e251518d7ce7bad395f00abaf6676c436d980e0077395cfaf76fc4889 not found: ID does not exist" Sep 10 21:00:34.538575 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.538552 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:34.541322 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.541302 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-755cfbd8df-pfbb5"] Sep 10 21:00:34.816428 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:34.816399 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7101912a-127c-4397-94bb-8dc604e2ca96" path="/var/lib/kubelet/pods/7101912a-127c-4397-94bb-8dc604e2ca96/volumes" Sep 10 21:00:40.535369 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:40.535335 2585 generic.go:356] "Generic (PLEG): container finished" podID="a662b93e-879c-4a28-af5e-1b23552d22d2" containerID="417c71ccee6ca721bab08cf38d07e013c691e73b6f46ec2e18890162db94009f" exitCode=0 Sep 10 21:00:40.535671 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:40.535409 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" event={"ID":"a662b93e-879c-4a28-af5e-1b23552d22d2","Type":"ContainerDied","Data":"417c71ccee6ca721bab08cf38d07e013c691e73b6f46ec2e18890162db94009f"} Sep 10 21:00:40.535741 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:40.535728 2585 scope.go:117] "RemoveContainer" containerID="417c71ccee6ca721bab08cf38d07e013c691e73b6f46ec2e18890162db94009f" Sep 10 21:00:41.542211 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:41.542176 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nph6h" event={"ID":"a662b93e-879c-4a28-af5e-1b23552d22d2","Type":"ContainerStarted","Data":"3f4770204065e41ffe511c0747f7ecc6f70584bff836858dbf01272a924b59e7"} Sep 10 21:00:52.649780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.649735 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54f8cdf484-l66t6" podUID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" containerName="console" containerID="cri-o://26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911" gracePeriod=15 Sep 10 21:00:52.884978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.884950 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:52.989416 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989354 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989419 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989447 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989470 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989491 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdfjm\" (UniqueName: \"kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989789 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989692 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989789 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989755 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config\") pod \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\" (UID: \"f8469c6f-7243-4c2d-a3bc-861f54a640e7\") " Sep 10 21:00:52.989913 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989882 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:52.990011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.989901 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:52.990096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.990068 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:52.990150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.990080 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-trusted-ca-bundle\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:52.990249 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.990120 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-oauth-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:52.990311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.990288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config" (OuterVolumeSpecName: "console-config") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:00:52.991642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.991597 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:00:52.991738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.991635 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm" (OuterVolumeSpecName: "kube-api-access-gdfjm") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "kube-api-access-gdfjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:00:52.991738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:52.991635 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8469c6f-7243-4c2d-a3bc-861f54a640e7" (UID: "f8469c6f-7243-4c2d-a3bc-861f54a640e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:00:53.091374 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.091351 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:53.091374 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.091374 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:53.091523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.091384 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8469c6f-7243-4c2d-a3bc-861f54a640e7-console-oauth-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:53.091523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.091393 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdfjm\" (UniqueName: \"kubernetes.io/projected/f8469c6f-7243-4c2d-a3bc-861f54a640e7-kube-api-access-gdfjm\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:53.091523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.091402 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8469c6f-7243-4c2d-a3bc-861f54a640e7-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:00:53.590773 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.590740 2585 generic.go:356] "Generic (PLEG): container finished" podID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" containerID="26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911" exitCode=0 Sep 10 21:00:53.590915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.590781 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8cdf484-l66t6" Sep 10 21:00:53.590915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.590808 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8cdf484-l66t6" event={"ID":"f8469c6f-7243-4c2d-a3bc-861f54a640e7","Type":"ContainerDied","Data":"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911"} Sep 10 21:00:53.590915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.590858 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8cdf484-l66t6" event={"ID":"f8469c6f-7243-4c2d-a3bc-861f54a640e7","Type":"ContainerDied","Data":"5e3eefcbe97783e7f48f66736ad90d067a482fc7c37fb507a445ee46aeaeb9b8"} Sep 10 21:00:53.590915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.590878 2585 scope.go:117] "RemoveContainer" containerID="26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911" Sep 10 21:00:53.599464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.599445 2585 scope.go:117] "RemoveContainer" containerID="26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911" Sep 10 21:00:53.599748 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:00:53.599729 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911\": container with ID starting with 26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911 not found: ID does not exist" containerID="26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911" Sep 10 21:00:53.599799 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.599756 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911"} err="failed to get container status \"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911\": rpc error: code = NotFound desc = could not find container \"26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911\": container with ID starting with 26694cbbb3cd2f1e2f8d5f28c9635c04a027d7b37cab9fcd744a93b3fa194911 not found: ID does not exist" Sep 10 21:00:53.614980 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.614959 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:53.617062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:53.617042 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54f8cdf484-l66t6"] Sep 10 21:00:54.817566 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:00:54.817522 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" path="/var/lib/kubelet/pods/f8469c6f-7243-4c2d-a3bc-861f54a640e7/volumes" Sep 10 21:01:23.036208 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036175 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036463 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="alertmanager" containerID="cri-o://5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" gracePeriod=120 Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036515 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-metric" containerID="cri-o://ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" gracePeriod=120 Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036570 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-web" containerID="cri-o://3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" gracePeriod=120 Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036578 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy" containerID="cri-o://48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" gracePeriod=120 Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036606 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="config-reloader" containerID="cri-o://20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" gracePeriod=120 Sep 10 21:01:23.036702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.036590 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="prom-label-proxy" containerID="cri-o://2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" gracePeriod=120 Sep 10 21:01:23.713533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713502 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" exitCode=0 Sep 10 21:01:23.713533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713524 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" exitCode=0 Sep 10 21:01:23.713533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713531 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" exitCode=0 Sep 10 21:01:23.713533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713536 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" exitCode=0 Sep 10 21:01:23.713801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713570 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2"} Sep 10 21:01:23.713801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713652 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a"} Sep 10 21:01:23.713801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713680 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69"} Sep 10 21:01:23.713801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:23.713705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f"} Sep 10 21:01:24.283408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.283377 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.311072 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311048 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311175 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311078 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311175 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311123 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311241 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311281 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311395 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311321 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311395 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311395 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311382 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311395 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:01:24.311538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311417 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zs9d\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311443 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311476 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.311786 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.311661 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:01:24.313165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.312002 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.313165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.312058 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume\") pod \"792683bb-1b80-401c-9bab-63746ab78e2c\" (UID: \"792683bb-1b80-401c-9bab-63746ab78e2c\") " Sep 10 21:01:24.313165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.312323 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-main-db\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.313165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.312340 2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-metrics-client-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.313165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.313123 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:01:24.314380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314343 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.314503 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314382 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d" (OuterVolumeSpecName: "kube-api-access-8zs9d") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "kube-api-access-8zs9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:01:24.314574 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314552 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.314682 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314652 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out" (OuterVolumeSpecName: "config-out") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:01:24.314835 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314801 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.314899 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.314839 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.315126 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.315110 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:01:24.315575 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.315558 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume" (OuterVolumeSpecName: "config-volume") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.319852 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.319739 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.324780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.324760 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config" (OuterVolumeSpecName: "web-config") pod "792683bb-1b80-401c-9bab-63746ab78e2c" (UID: "792683bb-1b80-401c-9bab-63746ab78e2c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413343 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413369 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-main-tls\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413379 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413388 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zs9d\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-kube-api-access-8zs9d\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413397 2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-cluster-tls-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413406 2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-web-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413414 2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/792683bb-1b80-401c-9bab-63746ab78e2c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413423 2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-config-volume\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413432 2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/792683bb-1b80-401c-9bab-63746ab78e2c-tls-assets\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413439 2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/792683bb-1b80-401c-9bab-63746ab78e2c-config-out\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.413697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.413446 2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/792683bb-1b80-401c-9bab-63746ab78e2c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:01:24.720941 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.720856 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" exitCode=0 Sep 10 21:01:24.720941 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.720881 2585 generic.go:356] "Generic (PLEG): container finished" podID="792683bb-1b80-401c-9bab-63746ab78e2c" containerID="3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" exitCode=0 Sep 10 21:01:24.721149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.720942 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65"} Sep 10 21:01:24.721149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.720959 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.721149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.721019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959"} Sep 10 21:01:24.721149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.721049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"792683bb-1b80-401c-9bab-63746ab78e2c","Type":"ContainerDied","Data":"085aa7d4cb60fea25dc350d24d77f91afe22e1d686f9c992651586bcbe8d1612"} Sep 10 21:01:24.721149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.721091 2585 scope.go:117] "RemoveContainer" containerID="2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" Sep 10 21:01:24.729549 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.729534 2585 scope.go:117] "RemoveContainer" containerID="ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" Sep 10 21:01:24.736523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.736506 2585 scope.go:117] "RemoveContainer" containerID="48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" Sep 10 21:01:24.743057 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.743041 2585 scope.go:117] "RemoveContainer" containerID="3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" Sep 10 21:01:24.747027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.747009 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:24.750992 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.750974 2585 scope.go:117] "RemoveContainer" containerID="20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" Sep 10 21:01:24.751225 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.751207 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:24.757534 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.757518 2585 scope.go:117] "RemoveContainer" containerID="5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" Sep 10 21:01:24.764229 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.764214 2585 scope.go:117] "RemoveContainer" containerID="aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf" Sep 10 21:01:24.770637 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.770621 2585 scope.go:117] "RemoveContainer" containerID="2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" Sep 10 21:01:24.770888 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.770872 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2\": container with ID starting with 2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2 not found: ID does not exist" containerID="2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" Sep 10 21:01:24.770945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.770894 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2"} err="failed to get container status \"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2\": rpc error: code = NotFound desc = could not find container \"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2\": container with ID starting with 2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2 not found: ID does not exist" Sep 10 21:01:24.770945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.770911 2585 scope.go:117] "RemoveContainer" containerID="ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" Sep 10 21:01:24.771160 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.771143 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65\": container with ID starting with ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65 not found: ID does not exist" containerID="ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" Sep 10 21:01:24.771195 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771174 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65"} err="failed to get container status \"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65\": rpc error: code = NotFound desc = could not find container \"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65\": container with ID starting with ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65 not found: ID does not exist" Sep 10 21:01:24.771195 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771189 2585 scope.go:117] "RemoveContainer" containerID="48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" Sep 10 21:01:24.771455 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.771435 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a\": container with ID starting with 48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a not found: ID does not exist" containerID="48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" Sep 10 21:01:24.771512 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771459 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a"} err="failed to get container status \"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a\": rpc error: code = NotFound desc = could not find container \"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a\": container with ID starting with 48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a not found: ID does not exist" Sep 10 21:01:24.771512 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771473 2585 scope.go:117] "RemoveContainer" containerID="3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" Sep 10 21:01:24.771888 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.771871 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959\": container with ID starting with 3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959 not found: ID does not exist" containerID="3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" Sep 10 21:01:24.771938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771892 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959"} err="failed to get container status \"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959\": rpc error: code = NotFound desc = could not find container \"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959\": container with ID starting with 3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959 not found: ID does not exist" Sep 10 21:01:24.771938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.771908 2585 scope.go:117] "RemoveContainer" containerID="20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" Sep 10 21:01:24.772143 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.772126 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69\": container with ID starting with 20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69 not found: ID does not exist" containerID="20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" Sep 10 21:01:24.772189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772149 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69"} err="failed to get container status \"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69\": rpc error: code = NotFound desc = could not find container \"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69\": container with ID starting with 20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69 not found: ID does not exist" Sep 10 21:01:24.772189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772162 2585 scope.go:117] "RemoveContainer" containerID="5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" Sep 10 21:01:24.772361 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.772346 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f\": container with ID starting with 5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f not found: ID does not exist" containerID="5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" Sep 10 21:01:24.772401 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772364 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f"} err="failed to get container status \"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f\": rpc error: code = NotFound desc = could not find container \"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f\": container with ID starting with 5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f not found: ID does not exist" Sep 10 21:01:24.772401 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772376 2585 scope.go:117] "RemoveContainer" containerID="aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf" Sep 10 21:01:24.772623 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:01:24.772590 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf\": container with ID starting with aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf not found: ID does not exist" containerID="aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf" Sep 10 21:01:24.772666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772630 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf"} err="failed to get container status \"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf\": rpc error: code = NotFound desc = could not find container \"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf\": container with ID starting with aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf not found: ID does not exist" Sep 10 21:01:24.772666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772645 2585 scope.go:117] "RemoveContainer" containerID="2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2" Sep 10 21:01:24.772843 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772827 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2"} err="failed to get container status \"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2\": rpc error: code = NotFound desc = could not find container \"2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2\": container with ID starting with 2f63ec9528796a303b4534527dbebd2edc372de99a8a4660f4e5da026b91afa2 not found: ID does not exist" Sep 10 21:01:24.772887 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772843 2585 scope.go:117] "RemoveContainer" containerID="ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65" Sep 10 21:01:24.773013 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.772996 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65"} err="failed to get container status \"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65\": rpc error: code = NotFound desc = could not find container \"ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65\": container with ID starting with ec6c9dee7666ace492ea7cb959a1436b80be0f61bfc6693d8a5641825bd33b65 not found: ID does not exist" Sep 10 21:01:24.773055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773015 2585 scope.go:117] "RemoveContainer" containerID="48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a" Sep 10 21:01:24.773223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773208 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a"} err="failed to get container status \"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a\": rpc error: code = NotFound desc = could not find container \"48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a\": container with ID starting with 48a3b509e1296f5bd005871fc46f05aec5bb12ccb1907f7c18a357718e66c49a not found: ID does not exist" Sep 10 21:01:24.773223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773223 2585 scope.go:117] "RemoveContainer" containerID="3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959" Sep 10 21:01:24.773448 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773429 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959"} err="failed to get container status \"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959\": rpc error: code = NotFound desc = could not find container \"3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959\": container with ID starting with 3b9f8cda37777049c3ea828cfc56133b92f49b677dd81256a74429f4fa513959 not found: ID does not exist" Sep 10 21:01:24.773489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773448 2585 scope.go:117] "RemoveContainer" containerID="20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69" Sep 10 21:01:24.773692 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773676 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69"} err="failed to get container status \"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69\": rpc error: code = NotFound desc = could not find container \"20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69\": container with ID starting with 20abaa5a554d027330116eba05fe03c4dd63b643ffcb0e2b7593f90a4ab07b69 not found: ID does not exist" Sep 10 21:01:24.773756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773692 2585 scope.go:117] "RemoveContainer" containerID="5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f" Sep 10 21:01:24.773940 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773923 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f"} err="failed to get container status \"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f\": rpc error: code = NotFound desc = could not find container \"5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f\": container with ID starting with 5ab42c5159621e22b61c25084a7e131d0e8c9872aab4857ad026d41f662c131f not found: ID does not exist" Sep 10 21:01:24.773940 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.773939 2585 scope.go:117] "RemoveContainer" containerID="aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf" Sep 10 21:01:24.774145 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.774127 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf"} err="failed to get container status \"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf\": rpc error: code = NotFound desc = could not find container \"aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf\": container with ID starting with aae1b8db33a4bc00bf1b65e60903b59436b2578eff50a6699291fbfa2853becf not found: ID does not exist" Sep 10 21:01:24.804672 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804649 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:24.804909 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804898 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7101912a-127c-4397-94bb-8dc604e2ca96" containerName="console" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804911 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7101912a-127c-4397-94bb-8dc604e2ca96" containerName="console" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804920 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="prom-label-proxy" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804926 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="prom-label-proxy" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804933 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804939 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804946 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-web" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804951 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-web" Sep 10 21:01:24.804955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804956 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-metric" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804961 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-metric" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804969 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="alertmanager" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804974 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="alertmanager" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804981 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="init-config-reloader" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.804987 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="init-config-reloader" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805001 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="config-reloader" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805006 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="config-reloader" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805013 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" containerName="console" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805018 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" containerName="console" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805065 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="prom-label-proxy" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805072 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7101912a-127c-4397-94bb-8dc604e2ca96" containerName="console" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805078 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-web" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805084 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy-metric" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805090 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="alertmanager" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805095 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="config-reloader" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805101 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" containerName="kube-rbac-proxy" Sep 10 21:01:24.805171 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.805106 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8469c6f-7243-4c2d-a3bc-861f54a640e7" containerName="console" Sep 10 21:01:24.809699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.809684 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.813802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.813780 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 21:01:24.813883 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.813841 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 21:01:24.814077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814063 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 21:01:24.814156 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814122 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 21:01:24.814201 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814188 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 21:01:24.814236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814225 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 21:01:24.814332 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814319 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 21:01:24.815010 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.814990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815025 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815062 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815087 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815324 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-config-out\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815404 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815369 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815458 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815710 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-web-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815697 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 21:01:24.815820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.815967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzn82\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-kube-api-access-qzn82\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.816020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.815971 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zc2w7\"" Sep 10 21:01:24.817463 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.817439 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792683bb-1b80-401c-9bab-63746ab78e2c" path="/var/lib/kubelet/pods/792683bb-1b80-401c-9bab-63746ab78e2c/volumes" Sep 10 21:01:24.828377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.824311 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 21:01:24.834345 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.834325 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:24.916849 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-config-out\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.916975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916856 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.916975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.916975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.916975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.916975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.916956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-web-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917221 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917221 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917221 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzn82\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-kube-api-access-qzn82\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917299 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917527 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917398 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917691 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.917944 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.917909 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.919664 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.919640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-config-out\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.919976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.919951 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.919990 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c72d390b-040e-48ef-94dd-3840575c250b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.919999 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.920018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.920028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.920666 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-web-config\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920758 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.920687 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.920793 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.920766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.921840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.921824 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c72d390b-040e-48ef-94dd-3840575c250b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:24.926705 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:24.926689 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzn82\" (UniqueName: \"kubernetes.io/projected/c72d390b-040e-48ef-94dd-3840575c250b-kube-api-access-qzn82\") pod \"alertmanager-main-0\" (UID: \"c72d390b-040e-48ef-94dd-3840575c250b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:25.119178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:25.119153 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 21:01:25.411532 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:01:25.411506 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72d390b_040e_48ef_94dd_3840575c250b.slice/crio-4ca82984508f04720f6e81790637c370ee416e8bd67c1a2827fb81b830ecb1a2 WatchSource:0}: Error finding container 4ca82984508f04720f6e81790637c370ee416e8bd67c1a2827fb81b830ecb1a2: Status 404 returned error can't find the container with id 4ca82984508f04720f6e81790637c370ee416e8bd67c1a2827fb81b830ecb1a2 Sep 10 21:01:25.412115 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:25.412090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 21:01:25.726483 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:25.726392 2585 generic.go:356] "Generic (PLEG): container finished" podID="c72d390b-040e-48ef-94dd-3840575c250b" containerID="851df19cf8895e3fc69d5ee5b15c7fae9cab4fd50567a205f5ee4ff6dc2c5464" exitCode=0 Sep 10 21:01:25.726663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:25.726481 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerDied","Data":"851df19cf8895e3fc69d5ee5b15c7fae9cab4fd50567a205f5ee4ff6dc2c5464"} Sep 10 21:01:25.726663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:25.726558 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"4ca82984508f04720f6e81790637c370ee416e8bd67c1a2827fb81b830ecb1a2"} Sep 10 21:01:26.734849 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734817 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"334447595f39d1bbaa2ca885314fe030e351e3c0e7652a71230a64f15d91492c"} Sep 10 21:01:26.735205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734883 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"7392c97efcca7b280890e5d6dd6a3a08c3c903299c41c7c67c087eae040d9bd5"} Sep 10 21:01:26.735205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"40bd2ac8c1d2fc4ab6b69e7fa5ffe1a435129aac8bb04c4372a028634e750bff"} Sep 10 21:01:26.735205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"1f9c57db21a3fbead065acc0f30c271753aafd6ff27a1ea3b7df1f6d0a2b81e5"} Sep 10 21:01:26.735205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734972 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"007481cb6cf2db5207b79dc7ab641f1ea434d8b3b83e92cda70ec2935b9a4664"} Sep 10 21:01:26.735205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.734997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c72d390b-040e-48ef-94dd-3840575c250b","Type":"ContainerStarted","Data":"dc73478dcee98bde1c80faaa7e4e975a5eaebf484b5f632751898773a85d30a1"} Sep 10 21:01:26.784175 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:26.784125 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.768741539 podStartE2EDuration="2.784110432s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.368893ms" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:01:26.783767316 +0000 UTC m=+170.492061701" watchObservedRunningTime="2026-09-10 21:01:26.784110432 +0000 UTC m=+170.492404818" Sep 10 21:01:58.840278 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.840244 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:01:58.849874 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.849846 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.891044 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.891017 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:01:58.960949 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.960918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8fq\" (UniqueName: \"kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.960957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.961049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.961079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.961126 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.961148 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:58.961260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:58.961180 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.061738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.061875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.061875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061767 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.061875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.061875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061825 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8fq\" (UniqueName: \"kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.061923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062591 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.062565 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.062664 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.062662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.062809 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.062785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.064337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.064312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.064422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.064336 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.074399 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.074378 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8fq\" (UniqueName: \"kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq\") pod \"console-5dd79cddf8-z69qz\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.159524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.159457 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:01:59.434383 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.434315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:01:59.437195 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:01:59.437161 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0282b8e_0aff_4402_ab4b_47809a7858d9.slice/crio-bdca99000b1fddb47e54150743760847ea6792ca356a70d0ab975929fa9af5d4 WatchSource:0}: Error finding container bdca99000b1fddb47e54150743760847ea6792ca356a70d0ab975929fa9af5d4: Status 404 returned error can't find the container with id bdca99000b1fddb47e54150743760847ea6792ca356a70d0ab975929fa9af5d4 Sep 10 21:01:59.862945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.862915 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd79cddf8-z69qz" event={"ID":"a0282b8e-0aff-4402-ab4b-47809a7858d9","Type":"ContainerStarted","Data":"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8"} Sep 10 21:01:59.863307 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.862963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd79cddf8-z69qz" event={"ID":"a0282b8e-0aff-4402-ab4b-47809a7858d9","Type":"ContainerStarted","Data":"bdca99000b1fddb47e54150743760847ea6792ca356a70d0ab975929fa9af5d4"} Sep 10 21:01:59.882877 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:01:59.882834 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dd79cddf8-z69qz" podStartSLOduration=1.8828199890000001 podStartE2EDuration="1.882819989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:01:59.88146211 +0000 UTC m=+203.589756525" watchObservedRunningTime="2026-09-10 21:01:59.882819989 +0000 UTC m=+203.591114375" Sep 10 21:02:09.159823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:09.159790 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:02:09.160173 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:09.159873 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:02:09.164504 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:09.164479 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:02:09.908658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:09.908587 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:02:09.971408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:09.971374 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:02:34.990153 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:34.990109 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f8c9b495-fwpmg" podUID="818e5b32-0413-4c64-bb05-02649bded1b5" containerName="console" containerID="cri-o://1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac" gracePeriod=15 Sep 10 21:02:35.230558 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.230533 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:02:35.333830 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333796 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333850 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333887 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333904 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nppm\" (UniqueName: \"kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333925 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333952 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.333997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.333983 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca\") pod \"818e5b32-0413-4c64-bb05-02649bded1b5\" (UID: \"818e5b32-0413-4c64-bb05-02649bded1b5\") " Sep 10 21:02:35.334544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.334391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:02:35.334544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.334413 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:02:35.334738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.334573 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca" (OuterVolumeSpecName: "service-ca") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:02:35.334738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.334637 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config" (OuterVolumeSpecName: "console-config") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:02:35.336053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.336023 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm" (OuterVolumeSpecName: "kube-api-access-4nppm") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "kube-api-access-4nppm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:02:35.336190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.336169 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:02:35.336411 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.336388 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "818e5b32-0413-4c64-bb05-02649bded1b5" (UID: "818e5b32-0413-4c64-bb05-02649bded1b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:02:35.435186 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435156 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-oauth-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435186 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435184 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-oauth-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435197 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nppm\" (UniqueName: \"kubernetes.io/projected/818e5b32-0413-4c64-bb05-02649bded1b5-kube-api-access-4nppm\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435210 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-trusted-ca-bundle\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435229 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-console-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435241 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/818e5b32-0413-4c64-bb05-02649bded1b5-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:35.435402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:35.435251 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/818e5b32-0413-4c64-bb05-02649bded1b5-console-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:02:36.011162 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.011128 2585 generic.go:356] "Generic (PLEG): container finished" podID="818e5b32-0413-4c64-bb05-02649bded1b5" containerID="1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac" exitCode=0 Sep 10 21:02:36.011645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.011200 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8c9b495-fwpmg" Sep 10 21:02:36.011645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.011207 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8c9b495-fwpmg" event={"ID":"818e5b32-0413-4c64-bb05-02649bded1b5","Type":"ContainerDied","Data":"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac"} Sep 10 21:02:36.011645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.011262 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8c9b495-fwpmg" event={"ID":"818e5b32-0413-4c64-bb05-02649bded1b5","Type":"ContainerDied","Data":"d9d4b9080a06b653f94937d03a28eedefc6837dbf7fae6fa5a9ee8a033ee2735"} Sep 10 21:02:36.011645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.011283 2585 scope.go:117] "RemoveContainer" containerID="1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac" Sep 10 21:02:36.019798 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.019784 2585 scope.go:117] "RemoveContainer" containerID="1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac" Sep 10 21:02:36.020031 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:02:36.020014 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac\": container with ID starting with 1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac not found: ID does not exist" containerID="1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac" Sep 10 21:02:36.020116 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.020037 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac"} err="failed to get container status \"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac\": rpc error: code = NotFound desc = could not find container \"1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac\": container with ID starting with 1f5b9d971f28a84a702b3abea2a0b483296110a4c7cb7254e2b20cc3280078ac not found: ID does not exist" Sep 10 21:02:36.032369 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.032349 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:02:36.037435 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.037416 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f8c9b495-fwpmg"] Sep 10 21:02:36.816865 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:36.816839 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="818e5b32-0413-4c64-bb05-02649bded1b5" path="/var/lib/kubelet/pods/818e5b32-0413-4c64-bb05-02649bded1b5/volumes" Sep 10 21:02:45.352738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.352707 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq"] Sep 10 21:02:45.353098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.353023 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818e5b32-0413-4c64-bb05-02649bded1b5" containerName="console" Sep 10 21:02:45.353098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.353036 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="818e5b32-0413-4c64-bb05-02649bded1b5" containerName="console" Sep 10 21:02:45.353098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.353093 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="818e5b32-0413-4c64-bb05-02649bded1b5" containerName="console" Sep 10 21:02:45.357426 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.357409 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.359948 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.359930 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 21:02:45.360107 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.360085 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-r9q2z\"" Sep 10 21:02:45.360730 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.360716 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 21:02:45.364977 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.364954 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq"] Sep 10 21:02:45.509328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.509302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.509447 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.509334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78d65\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-kube-api-access-78d65\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.610282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.610221 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.610282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.610253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78d65\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-kube-api-access-78d65\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.619200 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.619176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.619376 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.619356 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78d65\" (UniqueName: \"kubernetes.io/projected/5da9b378-11da-4368-8f40-d6f63f78a1a2-kube-api-access-78d65\") pod \"cert-manager-cainjector-559cdd8c87-zzmtq\" (UID: \"5da9b378-11da-4368-8f40-d6f63f78a1a2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.666803 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.666779 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" Sep 10 21:02:45.939338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:45.939312 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq"] Sep 10 21:02:45.941022 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:02:45.940999 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da9b378_11da_4368_8f40_d6f63f78a1a2.slice/crio-d177b6b73e7ecabf79c79b1f4047a115dfe6a8b5fc14483464e083aa372f8c96 WatchSource:0}: Error finding container d177b6b73e7ecabf79c79b1f4047a115dfe6a8b5fc14483464e083aa372f8c96: Status 404 returned error can't find the container with id d177b6b73e7ecabf79c79b1f4047a115dfe6a8b5fc14483464e083aa372f8c96 Sep 10 21:02:46.059869 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:46.059834 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" event={"ID":"5da9b378-11da-4368-8f40-d6f63f78a1a2","Type":"ContainerStarted","Data":"d177b6b73e7ecabf79c79b1f4047a115dfe6a8b5fc14483464e083aa372f8c96"} Sep 10 21:02:49.075363 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:49.075327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" event={"ID":"5da9b378-11da-4368-8f40-d6f63f78a1a2","Type":"ContainerStarted","Data":"127944cfceb831cd3ce5563bd1c81ebe3b042c48ebf706b01046feb0ae20549f"} Sep 10 21:02:49.104501 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:49.104400 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zzmtq" podStartSLOduration=1.207626337 podStartE2EDuration="4.104385144s" totalImagesPullingTime="2.896758807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:49.103604213 +0000 UTC m=+252.811898600" watchObservedRunningTime="2026-09-10 21:02:49.104385144 +0000 UTC m=+252.812679531" Sep 10 21:02:58.610907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.610872 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j82gq"] Sep 10 21:02:58.614418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.614401 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.616973 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.616953 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bf2jk\"" Sep 10 21:02:58.624380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.624356 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j82gq"] Sep 10 21:02:58.704067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.704038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-bound-sa-token\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.704203 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.704092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5t44\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-kube-api-access-z5t44\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.805098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.805067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-bound-sa-token\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.805225 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.805126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5t44\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-kube-api-access-z5t44\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.814265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.814236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-bound-sa-token\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.814441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.814420 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5t44\" (UniqueName: \"kubernetes.io/projected/ae23c4fd-8187-4d4c-b976-ef41f7865c72-kube-api-access-z5t44\") pod \"cert-manager-55b4d954d5-j82gq\" (UID: \"ae23c4fd-8187-4d4c-b976-ef41f7865c72\") " pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:58.923697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:58.923600 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j82gq" Sep 10 21:02:59.198189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:02:59.198041 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j82gq"] Sep 10 21:02:59.200855 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:02:59.200829 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae23c4fd_8187_4d4c_b976_ef41f7865c72.slice/crio-d2817ba60cbe4266f022cfd98535dbc627463d6117460710714fdccbfd558606 WatchSource:0}: Error finding container d2817ba60cbe4266f022cfd98535dbc627463d6117460710714fdccbfd558606: Status 404 returned error can't find the container with id d2817ba60cbe4266f022cfd98535dbc627463d6117460710714fdccbfd558606 Sep 10 21:03:00.119237 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:00.119206 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j82gq" event={"ID":"ae23c4fd-8187-4d4c-b976-ef41f7865c72","Type":"ContainerStarted","Data":"d5040c9031edfc707a984174a0f66f61502e614791902ac7eee08f107ffff4e7"} Sep 10 21:03:00.119237 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:00.119245 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j82gq" event={"ID":"ae23c4fd-8187-4d4c-b976-ef41f7865c72","Type":"ContainerStarted","Data":"d2817ba60cbe4266f022cfd98535dbc627463d6117460710714fdccbfd558606"} Sep 10 21:03:00.136535 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:00.136489 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-j82gq" podStartSLOduration=2.136475494 podStartE2EDuration="2.136475494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:00.135394206 +0000 UTC m=+263.843688592" watchObservedRunningTime="2026-09-10 21:03:00.136475494 +0000 UTC m=+263.844769880" Sep 10 21:03:18.217886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.217851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78"] Sep 10 21:03:18.221443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.221423 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.228545 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.228525 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 21:03:18.228675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.228566 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ldf8n\"" Sep 10 21:03:18.228675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.228578 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 21:03:18.228977 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.228961 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 21:03:18.230728 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.230714 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 21:03:18.244947 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.244924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78"] Sep 10 21:03:18.250657 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.250633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.250745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.250663 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62m9\" (UniqueName: \"kubernetes.io/projected/3ccc173c-d65c-4efc-8736-37bc6ebf0855-kube-api-access-z62m9\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.250745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.250716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-tmp\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.250817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.250758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-manifests\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.250817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.250778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351297 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62m9\" (UniqueName: \"kubernetes.io/projected/3ccc173c-d65c-4efc-8736-37bc6ebf0855-kube-api-access-z62m9\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351503 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351353 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-tmp\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351503 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-manifests\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351503 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.351947 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.351912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-manifests\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.353602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.353573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ccc173c-d65c-4efc-8736-37bc6ebf0855-tmp\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.353830 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.353813 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.353938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.353923 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ccc173c-d65c-4efc-8736-37bc6ebf0855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.360033 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.360014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62m9\" (UniqueName: \"kubernetes.io/projected/3ccc173c-d65c-4efc-8736-37bc6ebf0855-kube-api-access-z62m9\") pod \"opendatahub-operator-controller-manager-55d69f6bb5-lpv78\" (UID: \"3ccc173c-d65c-4efc-8736-37bc6ebf0855\") " pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.531968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.531936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:18.815083 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:18.815050 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccc173c_d65c_4efc_8736_37bc6ebf0855.slice/crio-f4f7c9c89bfa66e2d81c8123eeb83d238e38bd7e2128661ed4bc8e338bf4fdcd WatchSource:0}: Error finding container f4f7c9c89bfa66e2d81c8123eeb83d238e38bd7e2128661ed4bc8e338bf4fdcd: Status 404 returned error can't find the container with id f4f7c9c89bfa66e2d81c8123eeb83d238e38bd7e2128661ed4bc8e338bf4fdcd Sep 10 21:03:18.817746 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:18.817722 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78"] Sep 10 21:03:19.196013 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:19.195929 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" event={"ID":"3ccc173c-d65c-4efc-8736-37bc6ebf0855","Type":"ContainerStarted","Data":"f4f7c9c89bfa66e2d81c8123eeb83d238e38bd7e2128661ed4bc8e338bf4fdcd"} Sep 10 21:03:22.212663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:22.212629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" event={"ID":"3ccc173c-d65c-4efc-8736-37bc6ebf0855","Type":"ContainerStarted","Data":"388e781b7bb201a8f37c61ae69cfabf74fd92079d5c48192363c75414b95a877"} Sep 10 21:03:24.223196 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:24.223162 2585 generic.go:356] "Generic (PLEG): container finished" podID="3ccc173c-d65c-4efc-8736-37bc6ebf0855" containerID="388e781b7bb201a8f37c61ae69cfabf74fd92079d5c48192363c75414b95a877" exitCode=0 Sep 10 21:03:24.223703 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:24.223218 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" event={"ID":"3ccc173c-d65c-4efc-8736-37bc6ebf0855","Type":"ContainerDied","Data":"388e781b7bb201a8f37c61ae69cfabf74fd92079d5c48192363c75414b95a877"} Sep 10 21:03:25.229840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:25.229807 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" event={"ID":"3ccc173c-d65c-4efc-8736-37bc6ebf0855","Type":"ContainerStarted","Data":"1e207942651186c0f11a57867b164b9b339cf94c30e9164c89bc91cee60263c4"} Sep 10 21:03:25.230237 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:25.230024 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:25.252527 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:25.252485 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" podStartSLOduration=2.510028807 podStartE2EDuration="7.252470359s" totalImagesPullingTime="3.684754901s" totalInitContainerRuntime="1.057686651s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:25.250683435 +0000 UTC m=+288.958977821" watchObservedRunningTime="2026-09-10 21:03:25.252470359 +0000 UTC m=+288.960764744" Sep 10 21:03:35.442364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.442323 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7"] Sep 10 21:03:35.445851 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.445825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.448605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.448580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 21:03:35.449470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.449450 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 21:03:35.449573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.449482 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-c4kh2\"" Sep 10 21:03:35.449573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.449519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 21:03:35.449573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.449461 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 21:03:35.449776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.449461 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 21:03:35.458721 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.458697 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7"] Sep 10 21:03:35.594918 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.594885 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.595108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.594932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-manager-config\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.595108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.594960 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6w8z\" (UniqueName: \"kubernetes.io/projected/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-kube-api-access-j6w8z\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.595108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.595079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-metrics-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.696279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.696209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6w8z\" (UniqueName: \"kubernetes.io/projected/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-kube-api-access-j6w8z\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.696279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.696266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-metrics-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.696480 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.696292 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.696480 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.696316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-manager-config\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.697052 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.697030 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-manager-config\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.698749 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.698719 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-metrics-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.698841 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.698748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-cert\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.706489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.706468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6w8z\" (UniqueName: \"kubernetes.io/projected/cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c-kube-api-access-j6w8z\") pod \"lws-controller-manager-79fcf6dff7-zqcs7\" (UID: \"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c\") " pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:35.754971 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:35.754943 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:36.039999 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.039935 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7"] Sep 10 21:03:36.042586 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:36.042560 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0fc5d_1a1c_4ec3_ad2d_f03c95b70c2c.slice/crio-f6362994b6fed8f346b439ac7ffd12aea808c8184e2da2d43b24104ac2f6794b WatchSource:0}: Error finding container f6362994b6fed8f346b439ac7ffd12aea808c8184e2da2d43b24104ac2f6794b: Status 404 returned error can't find the container with id f6362994b6fed8f346b439ac7ffd12aea808c8184e2da2d43b24104ac2f6794b Sep 10 21:03:36.235960 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.235931 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d69f6bb5-lpv78" Sep 10 21:03:36.277242 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.277216 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" event={"ID":"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c","Type":"ContainerStarted","Data":"f6362994b6fed8f346b439ac7ffd12aea808c8184e2da2d43b24104ac2f6794b"} Sep 10 21:03:36.725598 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.725570 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:03:36.734570 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.734481 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:03:36.741188 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.741161 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:03:36.745969 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.745950 2585 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 21:03:36.748542 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:36.748521 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:03:39.292055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:39.292020 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" event={"ID":"cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c","Type":"ContainerStarted","Data":"b679393b568087555e3a982b9ad32ab58b3c0c6be00c78a33ea2bb9f54e8f3a2"} Sep 10 21:03:39.292472 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:39.292135 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:39.320284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:39.320240 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" podStartSLOduration=2.012847228 podStartE2EDuration="4.320227747s" totalImagesPullingTime="2.307380519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:39.319433873 +0000 UTC m=+303.027728258" watchObservedRunningTime="2026-09-10 21:03:39.320227747 +0000 UTC m=+303.028522133" Sep 10 21:03:44.785885 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.785852 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws"] Sep 10 21:03:44.789426 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.789402 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.792025 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.792005 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 21:03:44.792206 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.792188 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 21:03:44.792265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.792232 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r9gnc\"" Sep 10 21:03:44.792809 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.792794 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 21:03:44.805352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.805321 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws"] Sep 10 21:03:44.874981 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.874939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.874981 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.874984 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9541fd13-ec64-49cf-85bc-638adf10a876-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875151 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66zw\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-kube-api-access-f66zw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875369 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875369 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875325 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.875369 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.875352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9541fd13-ec64-49cf-85bc-638adf10a876-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976757 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976757 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976762 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976806 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9541fd13-ec64-49cf-85bc-638adf10a876-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9541fd13-ec64-49cf-85bc-638adf10a876-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976904 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.976990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.976985 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66zw\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-kube-api-access-f66zw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.977333 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.977169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.977333 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.977249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.977412 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.977360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.977482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.977463 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.977530 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.977500 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9541fd13-ec64-49cf-85bc-638adf10a876-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.979218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.979195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9541fd13-ec64-49cf-85bc-638adf10a876-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.979461 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.979442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9541fd13-ec64-49cf-85bc-638adf10a876-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:44.992230 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:44.992208 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:45.003397 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:45.003377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66zw\" (UniqueName: \"kubernetes.io/projected/9541fd13-ec64-49cf-85bc-638adf10a876-kube-api-access-f66zw\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws\" (UID: \"9541fd13-ec64-49cf-85bc-638adf10a876\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:45.100258 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:45.100170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:45.375907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:45.375875 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws"] Sep 10 21:03:45.377323 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:45.377293 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9541fd13_ec64_49cf_85bc_638adf10a876.slice/crio-49a236c53a64635d161d0b0908c95a3d7330776602d2c7919e81c190edb4c7a4 WatchSource:0}: Error finding container 49a236c53a64635d161d0b0908c95a3d7330776602d2c7919e81c190edb4c7a4: Status 404 returned error can't find the container with id 49a236c53a64635d161d0b0908c95a3d7330776602d2c7919e81c190edb4c7a4 Sep 10 21:03:45.379138 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:45.379123 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:03:46.324817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:46.324775 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" event={"ID":"9541fd13-ec64-49cf-85bc-638adf10a876","Type":"ContainerStarted","Data":"49a236c53a64635d161d0b0908c95a3d7330776602d2c7919e81c190edb4c7a4"} Sep 10 21:03:47.761220 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:47.761181 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:03:47.761461 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:47.761262 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:03:47.761461 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:47.761293 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:03:48.339540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.339501 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" event={"ID":"9541fd13-ec64-49cf-85bc-638adf10a876","Type":"ContainerStarted","Data":"cf45224f90ee1f00d60c5246416f4ceb0d34a8f57be44bfb8267b36a31f3955b"} Sep 10 21:03:48.362066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.362023 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" podStartSLOduration=1.980321043 podStartE2EDuration="4.36200722s" totalImagesPullingTime="2.381686177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:48.360719582 +0000 UTC m=+312.069013971" watchObservedRunningTime="2026-09-10 21:03:48.36200722 +0000 UTC m=+312.070301606" Sep 10 21:03:48.852235 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.852208 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7bl8x"] Sep 10 21:03:48.855647 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.855631 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:48.858270 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.858245 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 21:03:48.858365 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.858352 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h92vp\"" Sep 10 21:03:48.870284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.870257 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7bl8x"] Sep 10 21:03:48.912744 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.912721 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c469da6-d7ca-448a-af8c-d1409eb1203d-cert\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:48.912862 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:48.912753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27v4c\" (UniqueName: \"kubernetes.io/projected/2c469da6-d7ca-448a-af8c-d1409eb1203d-kube-api-access-27v4c\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.014002 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.013962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c469da6-d7ca-448a-af8c-d1409eb1203d-cert\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.014002 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.014005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27v4c\" (UniqueName: \"kubernetes.io/projected/2c469da6-d7ca-448a-af8c-d1409eb1203d-kube-api-access-27v4c\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.016275 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.016249 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c469da6-d7ca-448a-af8c-d1409eb1203d-cert\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.028431 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.024831 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27v4c\" (UniqueName: \"kubernetes.io/projected/2c469da6-d7ca-448a-af8c-d1409eb1203d-kube-api-access-27v4c\") pod \"odh-model-controller-7998c67dc4-7bl8x\" (UID: \"2c469da6-d7ca-448a-af8c-d1409eb1203d\") " pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.100747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.100719 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:49.105108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.105055 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:49.166730 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.166707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:49.344452 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.344420 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:49.345266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.345245 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws" Sep 10 21:03:49.434106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.434035 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7bl8x"] Sep 10 21:03:49.437302 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:49.437277 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c469da6_d7ca_448a_af8c_d1409eb1203d.slice/crio-b1531f3ade1e120100436d80b6e2c35c0c5a60628354cc648cdae9d9cbe1316e WatchSource:0}: Error finding container b1531f3ade1e120100436d80b6e2c35c0c5a60628354cc648cdae9d9cbe1316e: Status 404 returned error can't find the container with id b1531f3ade1e120100436d80b6e2c35c0c5a60628354cc648cdae9d9cbe1316e Sep 10 21:03:49.456580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.456555 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zwfzw"] Sep 10 21:03:49.461068 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.461052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:49.463265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.463243 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 21:03:49.463365 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.463344 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8j8l5\"" Sep 10 21:03:49.470968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.470946 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zwfzw"] Sep 10 21:03:49.517074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.517047 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjkf\" (UniqueName: \"kubernetes.io/projected/347b189d-6114-4d35-ba84-d44240c7e302-kube-api-access-6bjkf\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:49.517175 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.517085 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:49.618411 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.618369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjkf\" (UniqueName: \"kubernetes.io/projected/347b189d-6114-4d35-ba84-d44240c7e302-kube-api-access-6bjkf\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:49.618411 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.618417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:49.618720 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:03:49.618592 2585 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 21:03:49.618720 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:03:49.618688 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs podName:347b189d-6114-4d35-ba84-d44240c7e302 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:50.118666051 +0000 UTC m=+313.826960429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs") pod "model-serving-api-54d89bd5fc-zwfzw" (UID: "347b189d-6114-4d35-ba84-d44240c7e302") : secret "model-serving-api-tls" not found Sep 10 21:03:49.627046 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:49.627025 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjkf\" (UniqueName: \"kubernetes.io/projected/347b189d-6114-4d35-ba84-d44240c7e302-kube-api-access-6bjkf\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:50.123199 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.123158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:50.126584 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.126549 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347b189d-6114-4d35-ba84-d44240c7e302-tls-certs\") pod \"model-serving-api-54d89bd5fc-zwfzw\" (UID: \"347b189d-6114-4d35-ba84-d44240c7e302\") " pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:50.298465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.298431 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79fcf6dff7-zqcs7" Sep 10 21:03:50.350028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.349996 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" event={"ID":"2c469da6-d7ca-448a-af8c-d1409eb1203d","Type":"ContainerStarted","Data":"b1531f3ade1e120100436d80b6e2c35c0c5a60628354cc648cdae9d9cbe1316e"} Sep 10 21:03:50.371508 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.371482 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:50.646690 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:50.646659 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zwfzw"] Sep 10 21:03:50.648985 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:50.648957 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347b189d_6114_4d35_ba84_d44240c7e302.slice/crio-945a6157d54fad57385961c5ea13564764a2efd97c4dd177aa5c9f13e5a0b847 WatchSource:0}: Error finding container 945a6157d54fad57385961c5ea13564764a2efd97c4dd177aa5c9f13e5a0b847: Status 404 returned error can't find the container with id 945a6157d54fad57385961c5ea13564764a2efd97c4dd177aa5c9f13e5a0b847 Sep 10 21:03:51.358128 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:51.358083 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" event={"ID":"347b189d-6114-4d35-ba84-d44240c7e302","Type":"ContainerStarted","Data":"945a6157d54fad57385961c5ea13564764a2efd97c4dd177aa5c9f13e5a0b847"} Sep 10 21:03:54.374968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.374925 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" event={"ID":"347b189d-6114-4d35-ba84-d44240c7e302","Type":"ContainerStarted","Data":"2675fc8f976be93b549e740fd0b1d7b5b1df2a2c9f1b5e5577896ca88e7657ba"} Sep 10 21:03:54.375398 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.375030 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:03:54.376481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.376460 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" event={"ID":"2c469da6-d7ca-448a-af8c-d1409eb1203d","Type":"ContainerStarted","Data":"660af03bdfa18f7e2010d465af0d952151d990003eeff0ad76d5edbf9d40a173"} Sep 10 21:03:54.376562 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.376505 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:03:54.393780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.393739 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" podStartSLOduration=2.6105795499999997 podStartE2EDuration="5.393726073s" totalImagesPullingTime="2.783146523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:54.392242235 +0000 UTC m=+318.100536622" watchObservedRunningTime="2026-09-10 21:03:54.393726073 +0000 UTC m=+318.102020460" Sep 10 21:03:54.410678 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:54.410635 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" podStartSLOduration=2.46334362 podStartE2EDuration="6.41060638s" totalImagesPullingTime="3.94726276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:54.408902896 +0000 UTC m=+318.117197280" watchObservedRunningTime="2026-09-10 21:03:54.41060638 +0000 UTC m=+318.118900765" Sep 10 21:03:55.742249 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.742216 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv"] Sep 10 21:03:55.750028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.750004 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.754024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.754005 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 21:03:55.754134 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.754006 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 21:03:55.755350 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.755325 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vc6c8\"" Sep 10 21:03:55.757377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.757362 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 21:03:55.763216 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.763192 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv"] Sep 10 21:03:55.773665 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773640 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773744 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773744 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773825 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slldm\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-kube-api-access-slldm\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773825 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773896 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.773945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.773900 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.874962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.874932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.874962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.874962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.875167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.874979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.875167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.875004 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slldm\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-kube-api-access-slldm\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.875167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.875036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.875167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.875063 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.875167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.875086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.876139 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.876103 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.878080 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.878057 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.878276 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.878254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.878337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.878282 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.878337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.878293 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.883784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.883761 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slldm\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-kube-api-access-slldm\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:55.884380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:55.884364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a80e7f97-2f59-4963-8fe8-bac43d6b0f7e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cwlcv\" (UID: \"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:56.059603 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:56.059578 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:56.347900 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:56.347871 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv"] Sep 10 21:03:56.349588 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:03:56.349550 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80e7f97_2f59_4963_8fe8_bac43d6b0f7e.slice/crio-74ba573ea962a090d8ee5a0f98aa1a9e87e234e8617d93e35afa30a53a663f87 WatchSource:0}: Error finding container 74ba573ea962a090d8ee5a0f98aa1a9e87e234e8617d93e35afa30a53a663f87: Status 404 returned error can't find the container with id 74ba573ea962a090d8ee5a0f98aa1a9e87e234e8617d93e35afa30a53a663f87 Sep 10 21:03:56.387437 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:56.387408 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" event={"ID":"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e","Type":"ContainerStarted","Data":"74ba573ea962a090d8ee5a0f98aa1a9e87e234e8617d93e35afa30a53a663f87"} Sep 10 21:03:58.586806 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:58.586772 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:03:58.587058 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:58.586836 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:03:59.410313 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:59.410275 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" event={"ID":"a80e7f97-2f59-4963-8fe8-bac43d6b0f7e","Type":"ContainerStarted","Data":"85ba7e3792033869df25a0977f70953d7cde3118190fb3d0641b84212d280da9"} Sep 10 21:03:59.410505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:59.410480 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:03:59.412121 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:59.412090 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-cwlcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 21:03:59.412251 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:59.412157 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" podUID="a80e7f97-2f59-4963-8fe8-bac43d6b0f7e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:03:59.438524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:03:59.438474 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" podStartSLOduration=2.204166876 podStartE2EDuration="4.438462323s" totalImagesPullingTime="2.234295447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:59.438008451 +0000 UTC m=+323.146302837" watchObservedRunningTime="2026-09-10 21:03:59.438462323 +0000 UTC m=+323.146756708" Sep 10 21:04:00.416098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:04:00.416072 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cwlcv" Sep 10 21:04:05.383297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:04:05.383267 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7bl8x" Sep 10 21:04:05.385475 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:04:05.385456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zwfzw" Sep 10 21:05:17.057125 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:17.057092 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:05:42.076307 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.076242 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dd79cddf8-z69qz" podUID="a0282b8e-0aff-4402-ab4b-47809a7858d9" containerName="console" containerID="cri-o://ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8" gracePeriod=15 Sep 10 21:05:42.312099 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.312073 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:05:42.456141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456056 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456113 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456130 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456165 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456220 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456257 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n8fq\" (UniqueName: \"kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456287 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert\") pod \"a0282b8e-0aff-4402-ab4b-47809a7858d9\" (UID: \"a0282b8e-0aff-4402-ab4b-47809a7858d9\") " Sep 10 21:05:42.456578 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456519 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:05:42.456578 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456543 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:05:42.456715 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456599 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config" (OuterVolumeSpecName: "console-config") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:05:42.456958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.456933 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:05:42.458511 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.458474 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:05:42.458629 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.458509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq" (OuterVolumeSpecName: "kube-api-access-7n8fq") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "kube-api-access-7n8fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:05:42.458629 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.458556 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a0282b8e-0aff-4402-ab4b-47809a7858d9" (UID: "a0282b8e-0aff-4402-ab4b-47809a7858d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:05:42.557878 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557844 2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.557878 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557871 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-trusted-ca-bundle\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.557878 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557880 2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.558085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557888 2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-oauth-serving-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.558085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557896 2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-oauth-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.558085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557905 2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0282b8e-0aff-4402-ab4b-47809a7858d9-console-config\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.558085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.557913 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8fq\" (UniqueName: \"kubernetes.io/projected/a0282b8e-0aff-4402-ab4b-47809a7858d9-kube-api-access-7n8fq\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:05:42.883769 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.883734 2585 generic.go:356] "Generic (PLEG): container finished" podID="a0282b8e-0aff-4402-ab4b-47809a7858d9" containerID="ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8" exitCode=0 Sep 10 21:05:42.883920 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.883786 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dd79cddf8-z69qz" Sep 10 21:05:42.883920 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.883818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd79cddf8-z69qz" event={"ID":"a0282b8e-0aff-4402-ab4b-47809a7858d9","Type":"ContainerDied","Data":"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8"} Sep 10 21:05:42.883920 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.883869 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dd79cddf8-z69qz" event={"ID":"a0282b8e-0aff-4402-ab4b-47809a7858d9","Type":"ContainerDied","Data":"bdca99000b1fddb47e54150743760847ea6792ca356a70d0ab975929fa9af5d4"} Sep 10 21:05:42.883920 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.883889 2585 scope.go:117] "RemoveContainer" containerID="ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8" Sep 10 21:05:42.892380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.892363 2585 scope.go:117] "RemoveContainer" containerID="ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8" Sep 10 21:05:42.892678 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:05:42.892657 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8\": container with ID starting with ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8 not found: ID does not exist" containerID="ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8" Sep 10 21:05:42.892750 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.892690 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8"} err="failed to get container status \"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8\": rpc error: code = NotFound desc = could not find container \"ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8\": container with ID starting with ba7b47af0560119ea3d10afb7beb69e015e307972cccbdb6c182d940826924f8 not found: ID does not exist" Sep 10 21:05:42.907162 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.907139 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:05:42.912783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:42.912764 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dd79cddf8-z69qz"] Sep 10 21:05:44.817749 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:05:44.817709 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0282b8e-0aff-4402-ab4b-47809a7858d9" path="/var/lib/kubelet/pods/a0282b8e-0aff-4402-ab4b-47809a7858d9/volumes" Sep 10 21:06:03.121603 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.121524 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc"] Sep 10 21:06:03.122039 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.122021 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0282b8e-0aff-4402-ab4b-47809a7858d9" containerName="console" Sep 10 21:06:03.122095 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.122043 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0282b8e-0aff-4402-ab4b-47809a7858d9" containerName="console" Sep 10 21:06:03.122135 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.122123 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0282b8e-0aff-4402-ab4b-47809a7858d9" containerName="console" Sep 10 21:06:03.125067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.125048 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.129243 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.129222 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q2z2s\"" Sep 10 21:06:03.137949 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.137924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc"] Sep 10 21:06:03.226424 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226393 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqcb\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-kube-api-access-nsqcb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226865 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0894d6-a8cf-4228-98f4-6bdf22536405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226865 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226766 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.226865 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.226794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327599 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327599 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqcb\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-kube-api-access-nsqcb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0894d6-a8cf-4228-98f4-6bdf22536405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.327901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.327853 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.328254 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.328015 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.328254 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.328162 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.328345 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.328256 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.328345 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.328294 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.328507 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.328489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0894d6-a8cf-4228-98f4-6bdf22536405-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.330037 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.330015 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.330146 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.330044 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.337225 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.337182 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.337330 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.337276 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqcb\" (UniqueName: \"kubernetes.io/projected/8e0894d6-a8cf-4228-98f4-6bdf22536405-kube-api-access-nsqcb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-fg5pc\" (UID: \"8e0894d6-a8cf-4228-98f4-6bdf22536405\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.434550 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.434479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:03.716874 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.716846 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc"] Sep 10 21:06:03.718625 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:06:03.718589 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0894d6_a8cf_4228_98f4_6bdf22536405.slice/crio-567ead2311c2d253b4d2cdc06021630d8d600b2b8ed2d772742f7047bc9a0045 WatchSource:0}: Error finding container 567ead2311c2d253b4d2cdc06021630d8d600b2b8ed2d772742f7047bc9a0045: Status 404 returned error can't find the container with id 567ead2311c2d253b4d2cdc06021630d8d600b2b8ed2d772742f7047bc9a0045 Sep 10 21:06:03.720828 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.720787 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:06:03.720910 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.720891 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:06:03.720954 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.720937 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:06:03.981174 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.981087 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" event={"ID":"8e0894d6-a8cf-4228-98f4-6bdf22536405","Type":"ContainerStarted","Data":"e4e436fba6514edafdca71043c4f01833a04c6a70e414ac9aec0f0d259bc711f"} Sep 10 21:06:03.981174 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:03.981149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" event={"ID":"8e0894d6-a8cf-4228-98f4-6bdf22536405","Type":"ContainerStarted","Data":"567ead2311c2d253b4d2cdc06021630d8d600b2b8ed2d772742f7047bc9a0045"} Sep 10 21:06:04.006581 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:04.006535 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" podStartSLOduration=1.006520528 podStartE2EDuration="1.006520528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:04.005317212 +0000 UTC m=+447.713611602" watchObservedRunningTime="2026-09-10 21:06:04.006520528 +0000 UTC m=+447.714814913" Sep 10 21:06:04.435431 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:04.435400 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:04.439582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:04.439557 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:04.985677 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:04.985648 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:06:04.986566 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:06:04.986544 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:07:42.436956 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.436875 2585 generic.go:356] "Generic (PLEG): container finished" podID="8e0894d6-a8cf-4228-98f4-6bdf22536405" containerID="e4e436fba6514edafdca71043c4f01833a04c6a70e414ac9aec0f0d259bc711f" exitCode=0 Sep 10 21:07:42.437319 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.436952 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" event={"ID":"8e0894d6-a8cf-4228-98f4-6bdf22536405","Type":"ContainerDied","Data":"e4e436fba6514edafdca71043c4f01833a04c6a70e414ac9aec0f0d259bc711f"} Sep 10 21:07:42.437410 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.437393 2585 scope.go:117] "RemoveContainer" containerID="e4e436fba6514edafdca71043c4f01833a04c6a70e414ac9aec0f0d259bc711f" Sep 10 21:07:42.438408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.438377 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:07:42.438499 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.438459 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:07:42.438560 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:42.438515 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:07:43.443139 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:43.443107 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" event={"ID":"8e0894d6-a8cf-4228-98f4-6bdf22536405","Type":"ContainerStarted","Data":"3509d042dc95ef9d5d3a5c4e492c3574fddc77778abe7186e4acd18c2c16cf3e"} Sep 10 21:07:44.435288 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:44.435249 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:07:44.435505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:44.435484 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fg5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:44.435599 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:44.435577 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" podUID="8e0894d6-a8cf-4228-98f4-6bdf22536405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 21:07:45.435740 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:45.435708 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fg5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:45.436159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:45.435760 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" podUID="8e0894d6-a8cf-4228-98f4-6bdf22536405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 21:07:46.434899 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:46.434858 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fg5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:46.435075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:46.434917 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" podUID="8e0894d6-a8cf-4228-98f4-6bdf22536405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 21:07:47.435414 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:47.435378 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-fg5pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:47.435854 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:47.435430 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" podUID="8e0894d6-a8cf-4228-98f4-6bdf22536405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 10 21:07:48.438884 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:48.438853 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:07:48.439337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:48.439130 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:07:48.439729 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:48.439712 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-fg5pc" Sep 10 21:07:59.654671 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.654638 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt"] Sep 10 21:07:59.658458 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.658439 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.661248 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.661227 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 21:07:59.662266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.662241 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-njkdt\"" Sep 10 21:07:59.662361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.662247 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 21:07:59.662361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.662311 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 21:07:59.669783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.669760 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt"] Sep 10 21:07:59.692993 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.692963 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.693093 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.693020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.693156 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.693094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.693156 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.693109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.693156 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.693125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzctv\" (UniqueName: \"kubernetes.io/projected/52bc01d2-7ba9-4029-9915-bffdcdbb351c-kube-api-access-jzctv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.693268 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.693204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzctv\" (UniqueName: \"kubernetes.io/projected/52bc01d2-7ba9-4029-9915-bffdcdbb351c-kube-api-access-jzctv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794296 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794408 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794676 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794647 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794803 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794755 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.794803 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.794754 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.796407 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.796390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bc01d2-7ba9-4029-9915-bffdcdbb351c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.796795 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.796777 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bc01d2-7ba9-4029-9915-bffdcdbb351c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.809564 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.809539 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzctv\" (UniqueName: \"kubernetes.io/projected/52bc01d2-7ba9-4029-9915-bffdcdbb351c-kube-api-access-jzctv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wkdzt\" (UID: \"52bc01d2-7ba9-4029-9915-bffdcdbb351c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:07:59.969907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:07:59.969823 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:08:00.243846 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:00.243823 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt"] Sep 10 21:08:00.245578 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:08:00.245550 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bc01d2_7ba9_4029_9915_bffdcdbb351c.slice/crio-446d93b21377b60d533ab15d5ffcf8f3b089eadbcce34a37877669d97dc8641d WatchSource:0}: Error finding container 446d93b21377b60d533ab15d5ffcf8f3b089eadbcce34a37877669d97dc8641d: Status 404 returned error can't find the container with id 446d93b21377b60d533ab15d5ffcf8f3b089eadbcce34a37877669d97dc8641d Sep 10 21:08:00.522882 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:00.522842 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" event={"ID":"52bc01d2-7ba9-4029-9915-bffdcdbb351c","Type":"ContainerStarted","Data":"446d93b21377b60d533ab15d5ffcf8f3b089eadbcce34a37877669d97dc8641d"} Sep 10 21:08:01.530088 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:01.530059 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" event={"ID":"52bc01d2-7ba9-4029-9915-bffdcdbb351c","Type":"ContainerStarted","Data":"b832c955b04cc5dd3f63f2647ed71a48d477c1396442ba7483a4ba2dc5a75b8e"} Sep 10 21:08:01.549319 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:01.549272 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" podStartSLOduration=1.5329193 podStartE2EDuration="2.549252943s" totalImagesPullingTime="1.016333643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:01.548461149 +0000 UTC m=+565.256755534" watchObservedRunningTime="2026-09-10 21:08:01.549252943 +0000 UTC m=+565.257547331" Sep 10 21:08:09.969929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:09.969900 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:08:09.970366 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:09.969972 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:08:09.982486 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:09.982456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:08:10.583234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:10.583208 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wkdzt" Sep 10 21:08:13.038680 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.038645 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z"] Sep 10 21:08:13.042052 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.042034 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.044750 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.044729 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 21:08:13.060292 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.060271 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z"] Sep 10 21:08:13.111297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111267 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.111460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedcec28-0fa8-464c-8236-d8837a971d91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.111460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.111460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2x6c\" (UniqueName: \"kubernetes.io/projected/bedcec28-0fa8-464c-8236-d8837a971d91-kube-api-access-v2x6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.111460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.111604 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.111520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212544 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212580 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2x6c\" (UniqueName: \"kubernetes.io/projected/bedcec28-0fa8-464c-8236-d8837a971d91-kube-api-access-v2x6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212606 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212651 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.212826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.212711 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedcec28-0fa8-464c-8236-d8837a971d91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.213046 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.213021 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.213087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.213034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.213135 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.213111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.214956 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.214938 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedcec28-0fa8-464c-8236-d8837a971d91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.215154 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.215138 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedcec28-0fa8-464c-8236-d8837a971d91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.221229 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.221208 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2x6c\" (UniqueName: \"kubernetes.io/projected/bedcec28-0fa8-464c-8236-d8837a971d91-kube-api-access-v2x6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gwf2z\" (UID: \"bedcec28-0fa8-464c-8236-d8837a971d91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.352150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.352060 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:13.627560 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:13.627489 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z"] Sep 10 21:08:13.630975 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:08:13.630950 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedcec28_0fa8_464c_8236_d8837a971d91.slice/crio-ab4ca91257c35da51cad17b65fdf1176dfe3d38ae299e6c45644fdfda2b1b912 WatchSource:0}: Error finding container ab4ca91257c35da51cad17b65fdf1176dfe3d38ae299e6c45644fdfda2b1b912: Status 404 returned error can't find the container with id ab4ca91257c35da51cad17b65fdf1176dfe3d38ae299e6c45644fdfda2b1b912 Sep 10 21:08:14.593026 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:14.592987 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" event={"ID":"bedcec28-0fa8-464c-8236-d8837a971d91","Type":"ContainerStarted","Data":"05517ce6edec298c7d3d143133a1b08198b560dac6a0030451d1ef83c7641c98"} Sep 10 21:08:14.593443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:14.593041 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" event={"ID":"bedcec28-0fa8-464c-8236-d8837a971d91","Type":"ContainerStarted","Data":"ab4ca91257c35da51cad17b65fdf1176dfe3d38ae299e6c45644fdfda2b1b912"} Sep 10 21:08:14.617230 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:14.617176 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" podStartSLOduration=1.4278646689999999 podStartE2EDuration="1.617159435s" totalImagesPullingTime="189.294766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:14.614183508 +0000 UTC m=+578.322477893" watchObservedRunningTime="2026-09-10 21:08:14.617159435 +0000 UTC m=+578.325453823" Sep 10 21:08:23.352885 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:23.352845 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:23.353306 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:23.352919 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:23.365989 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:23.365967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:23.652557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:23.652474 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gwf2z" Sep 10 21:08:36.766308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:36.766263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:08:36.771067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:36.771041 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:08:36.774441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:36.774423 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:08:36.779028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:36.779010 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:08:42.838672 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.838638 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv"] Sep 10 21:08:42.842752 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.842728 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.844931 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.844912 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 21:08:42.851356 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.851332 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv"] Sep 10 21:08:42.858344 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858319 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.858443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.858443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqw4r\" (UniqueName: \"kubernetes.io/projected/09633c8d-94cc-499d-88e6-bae86a394555-kube-api-access-qqw4r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.858553 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858521 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09633c8d-94cc-499d-88e6-bae86a394555-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.858634 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858562 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.858634 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.858595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959665 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09633c8d-94cc-499d-88e6-bae86a394555-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959779 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959779 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959779 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.959933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.959846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqw4r\" (UniqueName: \"kubernetes.io/projected/09633c8d-94cc-499d-88e6-bae86a394555-kube-api-access-qqw4r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.960204 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.960175 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.960308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.960185 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.960308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.960246 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.961951 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.961933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09633c8d-94cc-499d-88e6-bae86a394555-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.962112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.962092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09633c8d-94cc-499d-88e6-bae86a394555-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:42.968780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:42.968758 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqw4r\" (UniqueName: \"kubernetes.io/projected/09633c8d-94cc-499d-88e6-bae86a394555-kube-api-access-qqw4r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mxnfv\" (UID: \"09633c8d-94cc-499d-88e6-bae86a394555\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:43.154115 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:43.154038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:43.426183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:43.426155 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv"] Sep 10 21:08:43.428262 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:08:43.428219 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09633c8d_94cc_499d_88e6_bae86a394555.slice/crio-a536f22f36108385c713bc67295f83523aab62dd55ae9d4fb1774eb904992573 WatchSource:0}: Error finding container a536f22f36108385c713bc67295f83523aab62dd55ae9d4fb1774eb904992573: Status 404 returned error can't find the container with id a536f22f36108385c713bc67295f83523aab62dd55ae9d4fb1774eb904992573 Sep 10 21:08:43.736377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:43.736296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" event={"ID":"09633c8d-94cc-499d-88e6-bae86a394555","Type":"ContainerStarted","Data":"fa04e622ee25fb3029a3bae4fb8b23b638baa1aa9914e903982ccb8f6e3e6e73"} Sep 10 21:08:43.736377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:43.736357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" event={"ID":"09633c8d-94cc-499d-88e6-bae86a394555","Type":"ContainerStarted","Data":"a536f22f36108385c713bc67295f83523aab62dd55ae9d4fb1774eb904992573"} Sep 10 21:08:43.754582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:43.754536 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" podStartSLOduration=1.561818842 podStartE2EDuration="1.754522069s" totalImagesPullingTime="192.703227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:43.753109009 +0000 UTC m=+607.461403394" watchObservedRunningTime="2026-09-10 21:08:43.754522069 +0000 UTC m=+607.462816455" Sep 10 21:08:53.154311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:53.154280 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:53.154687 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:53.154361 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:53.166573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:53.166551 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:08:53.800351 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:08:53.800317 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mxnfv" Sep 10 21:10:01.046091 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.046058 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:10:01.049749 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.049726 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.053355 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.053332 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-76b274-openshift-default-dockercfg-fq5qd\"" Sep 10 21:10:01.066854 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.066829 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:10:01.136505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136673 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136673 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136894 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136813 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136894 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136867 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.136979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.136962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfj5\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237443 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237452 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfj5\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237941 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.237941 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237870 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.238051 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.237970 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.238105 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.238057 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.238177 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.238158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.238232 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.238204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.244628 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.238603 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.244628 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.238723 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.245444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.245416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.246030 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.246007 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.247804 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.247784 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfj5\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.248122 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.248105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token\") pod \"e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.360267 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.360182 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:01.637950 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.637927 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:10:01.639900 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:10:01.639874 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957fdd18_3e09_4eb7_b737_380fb84874ce.slice/crio-bdda57985e736587616b79c36ef8914defb54c2b2db6b93af1c1c576d4fa860b WatchSource:0}: Error finding container bdda57985e736587616b79c36ef8914defb54c2b2db6b93af1c1c576d4fa860b: Status 404 returned error can't find the container with id bdda57985e736587616b79c36ef8914defb54c2b2db6b93af1c1c576d4fa860b Sep 10 21:10:01.642473 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.642433 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:01.642571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.642498 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:01.642571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:01.642529 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:02.117192 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:02.117152 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerStarted","Data":"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c"} Sep 10 21:10:02.117686 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:02.117207 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerStarted","Data":"bdda57985e736587616b79c36ef8914defb54c2b2db6b93af1c1c576d4fa860b"} Sep 10 21:10:02.139747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:02.139705 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podStartSLOduration=1.139689363 podStartE2EDuration="1.139689363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:02.137907357 +0000 UTC m=+685.846201743" watchObservedRunningTime="2026-09-10 21:10:02.139689363 +0000 UTC m=+685.847983749" Sep 10 21:10:02.360530 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:02.360486 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:02.365592 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:02.365568 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:03.122365 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:03.122337 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:03.123419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:03.123399 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:09.351062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.351012 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:10:09.354688 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.354664 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.357094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.357072 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d9fd5e6e-openshift-default-dockercfg-2vwbm\"" Sep 10 21:10:09.367101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.367078 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:10:09.514279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514286 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514416 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514460 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95ng\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.514694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.514580 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615510 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95ng\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615510 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615634 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615661 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615712 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615739 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.615778 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.616121 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.615989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.616418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.616388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.616637 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.616410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.616776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.616714 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.617154 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.617124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.617243 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.617183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.619671 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.619641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.619881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.619853 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.626348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.626322 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95ng\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.626540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.626493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token\") pod \"e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.667573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.667541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:09.971509 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.971424 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:10:09.999087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.999024 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:09.999291 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.999267 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:09.999437 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:09.999416 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:10.156820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.156782 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerStarted","Data":"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6"} Sep 10 21:10:10.156979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.156839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerStarted","Data":"7e0e4833925fcc06f0b3f5e89c2d2e6bf047e1ca22fb64a147fb51ee2b23820d"} Sep 10 21:10:10.183057 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.182976 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podStartSLOduration=1.182953882 podStartE2EDuration="1.182953882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:10.1787167 +0000 UTC m=+693.887011087" watchObservedRunningTime="2026-09-10 21:10:10.182953882 +0000 UTC m=+693.891248268" Sep 10 21:10:10.420401 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.420362 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:10:10.424211 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.424190 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.426441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.426419 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f96a7a-openshift-default-dockercfg-qrjkm\"" Sep 10 21:10:10.434163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.434142 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:10:10.526273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526462 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526289 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526462 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526466 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526538 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.526763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.526709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mndd\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.627728 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.627854 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627739 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.627927 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.627927 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.627987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mndd\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628216 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628216 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628457 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628603 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628581 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628655 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.628806 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.628702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.630887 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.630867 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.630981 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.630920 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.636361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.636334 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.637049 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.637027 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mndd\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd\") pod \"e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:10.667743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.667721 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:10.735429 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:10.735363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:11.036422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.036397 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:10:11.038167 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:10:11.038145 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab11ec1_26dc_4db2_97f4_77c85667517f.slice/crio-d81474d99cc8632d69425fb35e1bb958bf577f13d162f1be877189edc5aa8563 WatchSource:0}: Error finding container d81474d99cc8632d69425fb35e1bb958bf577f13d162f1be877189edc5aa8563: Status 404 returned error can't find the container with id d81474d99cc8632d69425fb35e1bb958bf577f13d162f1be877189edc5aa8563 Sep 10 21:10:11.040266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.040236 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:11.040317 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.040298 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:11.040351 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.040326 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:11.165155 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.165119 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerStarted","Data":"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6"} Sep 10 21:10:11.165300 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.165203 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerStarted","Data":"d81474d99cc8632d69425fb35e1bb958bf577f13d162f1be877189edc5aa8563"} Sep 10 21:10:11.197909 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.197853 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podStartSLOduration=1.197831069 podStartE2EDuration="1.197831069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:11.194913858 +0000 UTC m=+694.903208238" watchObservedRunningTime="2026-09-10 21:10:11.197831069 +0000 UTC m=+694.906125458" Sep 10 21:10:11.673298 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.673270 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:11.735791 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:11.735748 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:12.169631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:12.169588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:12.170755 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:12.170737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:12.740849 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:12.740820 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:13.176088 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:13.176055 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:13.181223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:13.181019 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:18.250104 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:18.248862 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:44602->10.133.0.37:15021: read: connection reset by peer" start-of-body= Sep 10 21:10:18.250104 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:18.248923 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:44602->10.133.0.37:15021: read: connection reset by peer" Sep 10 21:10:19.219488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.219453 2585 generic.go:356] "Generic (PLEG): container finished" podID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerID="8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c" exitCode=0 Sep 10 21:10:19.219710 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.219500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerDied","Data":"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c"} Sep 10 21:10:19.219876 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.219860 2585 scope.go:117] "RemoveContainer" containerID="8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c" Sep 10 21:10:19.220940 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.220909 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:19.221030 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.220979 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:19.221030 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:19.221005 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:20.231875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:20.231839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerStarted","Data":"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d"} Sep 10 21:10:20.360475 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:20.360439 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:20.360688 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:20.360669 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:20.360739 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:20.360711 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:21.360958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:21.360927 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:21.361318 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:21.360996 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:22.361104 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:22.361073 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:22.361479 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:22.361123 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:23.360624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:23.360576 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:23.360824 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:23.360650 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:24.360865 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:24.360827 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:24.361253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:24.360882 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:25.360897 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:25.360853 2585 patch_prober.go:28] interesting pod/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 21:10:25.361481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:25.361448 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 21:10:26.365477 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:26.365442 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:26.365888 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:26.365867 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:26.366534 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:26.366514 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:10:29.520377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.520153 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:10:29.530886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.530859 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.538492 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.538466 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:10:29.540016 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.539993 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-31e7cb-openshift-default-dockercfg-j5gvx\"" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.645791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.645836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.645887 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.645950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.645977 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4n5\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.646002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.646052 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.646114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.646633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.646136 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747218 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747306 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747331 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747400 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.747582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4n5\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747750 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747792 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.747923 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.748014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.748266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.748174 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.750316 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.750296 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.750575 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.750555 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.756686 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.756665 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.757080 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.757063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4n5\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5\") pod \"e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:29.848066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:29.848030 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:30.171361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.171334 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:10:30.173583 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:10:30.173552 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3367312_ecd3_41d3_bd98_b85b459771b8.slice/crio-8bd537ee556da3601f6999f4bf243d49f124ec24af014ae791ee825728677905 WatchSource:0}: Error finding container 8bd537ee556da3601f6999f4bf243d49f124ec24af014ae791ee825728677905: Status 404 returned error can't find the container with id 8bd537ee556da3601f6999f4bf243d49f124ec24af014ae791ee825728677905 Sep 10 21:10:30.186890 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.181051 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:30.186890 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.181132 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:30.186890 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.181176 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:30.290949 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.290910 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerStarted","Data":"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263"} Sep 10 21:10:30.291112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.290973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerStarted","Data":"8bd537ee556da3601f6999f4bf243d49f124ec24af014ae791ee825728677905"} Sep 10 21:10:30.312574 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.312486 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podStartSLOduration=1.312460886 podStartE2EDuration="1.312460886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:30.310063849 +0000 UTC m=+714.018358264" watchObservedRunningTime="2026-09-10 21:10:30.312460886 +0000 UTC m=+714.020755276" Sep 10 21:10:30.849033 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:30.848999 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:31.854957 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:31.854930 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:32.303268 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:32.303233 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:32.304352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:32.304333 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:34.317201 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.317161 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerID="07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6" exitCode=0 Sep 10 21:10:34.317873 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.317246 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerDied","Data":"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6"} Sep 10 21:10:34.317873 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.317801 2585 scope.go:117] "RemoveContainer" containerID="07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6" Sep 10 21:10:34.319192 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319156 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:34.319310 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319242 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:34.319310 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319268 2585 generic.go:356] "Generic (PLEG): container finished" podID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerID="53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6" exitCode=0 Sep 10 21:10:34.319310 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319288 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:34.319489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319366 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerDied","Data":"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6"} Sep 10 21:10:34.319921 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.319899 2585 scope.go:117] "RemoveContainer" containerID="53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6" Sep 10 21:10:34.321162 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.321122 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:34.321269 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.321189 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:34.321269 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:34.321220 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:35.326337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.326306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerStarted","Data":"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1"} Sep 10 21:10:35.328445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.328418 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerStarted","Data":"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6"} Sep 10 21:10:35.668055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.667968 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:35.668218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.668171 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:35.668288 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.668226 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:35.736507 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.736474 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:35.736708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.736684 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:35.736754 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:35.736725 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:36.668440 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:36.668393 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:36.668847 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:36.668457 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:36.736509 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:36.736478 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:36.736677 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:36.736528 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:37.034128 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.034098 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t"] Sep 10 21:10:37.037523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.037506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:37.039962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.039938 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:10:37.040129 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.040114 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:10:37.045650 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.045628 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t"] Sep 10 21:10:37.127531 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.127491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t\" (UID: \"6d10c7a7-0a8f-472e-b26c-64b150c9e09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:37.228830 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.228787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t\" (UID: \"6d10c7a7-0a8f-472e-b26c-64b150c9e09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:37.229466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.229444 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t\" (UID: \"6d10c7a7-0a8f-472e-b26c-64b150c9e09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:37.348761 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.348681 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:37.667955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.667880 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:37.667955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.667937 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:37.692050 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.692019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t"] Sep 10 21:10:37.706636 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:10:37.704182 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d10c7a7_0a8f_472e_b26c_64b150c9e09c.slice/crio-324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e WatchSource:0}: Error finding container 324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e: Status 404 returned error can't find the container with id 324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e Sep 10 21:10:37.707123 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.706728 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:10:37.736119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.736084 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:37.736265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:37.736142 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:38.347777 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:38.347741 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" event={"ID":"6d10c7a7-0a8f-472e-b26c-64b150c9e09c","Type":"ContainerStarted","Data":"324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e"} Sep 10 21:10:38.668126 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:38.668052 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:38.668126 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:38.668113 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:38.736279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:38.736241 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:38.736658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:38.736322 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:39.668542 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:39.668508 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:39.668751 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:39.668567 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:39.735885 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:39.735845 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:39.736062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:39.735899 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:40.667923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:40.667887 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 21:10:40.668478 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:40.667953 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 21:10:40.735810 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:40.735776 2585 patch_prober.go:28] interesting pod/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 21:10:40.735988 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:40.735837 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 21:10:41.672439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.672364 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:41.672826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.672664 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:41.673241 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.673225 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:41.741311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.741277 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:41.741851 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.741820 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:41.742583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:41.742566 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:10:42.372311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:42.372278 2585 generic.go:356] "Generic (PLEG): container finished" podID="6d10c7a7-0a8f-472e-b26c-64b150c9e09c" containerID="27a5ce6de107e8bbe6892cacfadb30c02cf441611d0af13ab5418b5effa6fbac" exitCode=0 Sep 10 21:10:42.372454 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:42.372317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" event={"ID":"6d10c7a7-0a8f-472e-b26c-64b150c9e09c","Type":"ContainerDied","Data":"27a5ce6de107e8bbe6892cacfadb30c02cf441611d0af13ab5418b5effa6fbac"} Sep 10 21:10:43.550310 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:43.550288 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:43.600605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:43.600560 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca\") pod \"6d10c7a7-0a8f-472e-b26c-64b150c9e09c\" (UID: \"6d10c7a7-0a8f-472e-b26c-64b150c9e09c\") " Sep 10 21:10:43.600920 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:43.600892 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d10c7a7-0a8f-472e-b26c-64b150c9e09c" (UID: "6d10c7a7-0a8f-472e-b26c-64b150c9e09c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:10:43.601050 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:43.601028 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d10c7a7-0a8f-472e-b26c-64b150c9e09c-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:44.390004 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:44.389974 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" Sep 10 21:10:44.390379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:44.390003 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t" event={"ID":"6d10c7a7-0a8f-472e-b26c-64b150c9e09c","Type":"ContainerDied","Data":"324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e"} Sep 10 21:10:44.390379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:44.390352 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324d7555b42289c4ace61e5945fde44fc71f557d17fb0f789824f04bd6d0060e" Sep 10 21:10:51.435002 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.434923 2585 generic.go:356] "Generic (PLEG): container finished" podID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerID="e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263" exitCode=0 Sep 10 21:10:51.435399 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.434998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerDied","Data":"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263"} Sep 10 21:10:51.435519 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.435504 2585 scope.go:117] "RemoveContainer" containerID="e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263" Sep 10 21:10:51.436910 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.436882 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:51.437001 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.436952 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:51.437001 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:51.436983 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:52.443958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:52.443913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerStarted","Data":"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9"} Sep 10 21:10:52.849067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:52.849039 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:52.849334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:52.849305 2585 patch_prober.go:28] interesting pod/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 21:10:52.849444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:52.849355 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 21:10:53.848868 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:53.848836 2585 patch_prober.go:28] interesting pod/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 21:10:53.849222 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:53.848897 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 21:10:54.072747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.072704 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t"] Sep 10 21:10:54.078588 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.078557 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d9fd5e6e-cjl5t"] Sep 10 21:10:54.200285 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.200188 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:10:54.206895 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.200780 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" containerID="cri-o://40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6" gracePeriod=30 Sep 10 21:10:54.823893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.823850 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d10c7a7-0a8f-472e-b26c-64b150c9e09c" path="/var/lib/kubelet/pods/6d10c7a7-0a8f-472e-b26c-64b150c9e09c/volumes" Sep 10 21:10:54.848943 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.848912 2585 patch_prober.go:28] interesting pod/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 21:10:54.849309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:54.848963 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 21:10:55.154746 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.154664 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:10:55.155158 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.155139 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d10c7a7-0a8f-472e-b26c-64b150c9e09c" containerName="revoke-keys" Sep 10 21:10:55.155248 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.155160 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d10c7a7-0a8f-472e-b26c-64b150c9e09c" containerName="revoke-keys" Sep 10 21:10:55.155303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.155246 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d10c7a7-0a8f-472e-b26c-64b150c9e09c" containerName="revoke-keys" Sep 10 21:10:55.161748 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.161725 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.164551 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.164529 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bbddbd71-gw-openshift-default-dockercfg-wfh89\"" Sep 10 21:10:55.181464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.181414 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:10:55.329479 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329437 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329575 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpk2g\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329657 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329679 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329928 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329928 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.329928 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.329828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.430990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.430903 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.430994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431033 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpk2g\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431379 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431589 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431589 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431562 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431712 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431659 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431749 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431718 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.431886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.431864 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.433452 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.433431 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.433682 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.433664 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.444660 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.444639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpk2g\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.444660 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.444658 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token\") pod \"e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.471880 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.471861 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:55.795840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.791207 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:10:55.795840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.794737 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:55.795840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.794812 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:55.795840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.795097 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:10:55.848576 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.848546 2585 patch_prober.go:28] interesting pod/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 21:10:55.848689 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:55.848601 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 21:10:56.465332 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.465295 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerStarted","Data":"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605"} Sep 10 21:10:56.465768 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.465348 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerStarted","Data":"5c685d62c94021cd16c47c267c5fe9fa0ff672dc5f661bf1cfcbd8bd528f8af2"} Sep 10 21:10:56.472266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.472241 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:56.488642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.488603 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:56.498793 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.498290 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podStartSLOduration=1.498271019 podStartE2EDuration="1.498271019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:56.497001616 +0000 UTC m=+740.205296006" watchObservedRunningTime="2026-09-10 21:10:56.498271019 +0000 UTC m=+740.206565406" Sep 10 21:10:56.683009 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.682974 2585 patch_prober.go:28] interesting pod/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 21:10:56.683210 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.683040 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:10:56.849027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.848995 2585 patch_prober.go:28] interesting pod/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 21:10:56.849195 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:56.849045 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 21:10:57.470701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:57.470667 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:57.471726 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:57.471704 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:10:57.852163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:57.852135 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:57.852405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:57.852383 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:57.852941 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:57.852923 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:10:59.475409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.475385 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:59.482412 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.482388 2585 generic.go:356] "Generic (PLEG): container finished" podID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerID="40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6" exitCode=0 Sep 10 21:10:59.482533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.482429 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" Sep 10 21:10:59.482533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.482474 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerDied","Data":"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6"} Sep 10 21:10:59.482533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.482525 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk" event={"ID":"ce796f61-8e9d-4e04-97c7-6816c0c8a93c","Type":"ContainerDied","Data":"7e0e4833925fcc06f0b3f5e89c2d2e6bf047e1ca22fb64a147fb51ee2b23820d"} Sep 10 21:10:59.482682 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.482553 2585 scope.go:117] "RemoveContainer" containerID="40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6" Sep 10 21:10:59.493660 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.493641 2585 scope.go:117] "RemoveContainer" containerID="53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6" Sep 10 21:10:59.502945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.502927 2585 scope.go:117] "RemoveContainer" containerID="40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6" Sep 10 21:10:59.503245 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:10:59.503197 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6\": container with ID starting with 40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6 not found: ID does not exist" containerID="40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6" Sep 10 21:10:59.503313 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.503253 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6"} err="failed to get container status \"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6\": rpc error: code = NotFound desc = could not find container \"40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6\": container with ID starting with 40907e0f06d3ca02f5c69681a4000bb76cc8e5e95c380976ff90a825e63b53a6 not found: ID does not exist" Sep 10 21:10:59.503313 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.503272 2585 scope.go:117] "RemoveContainer" containerID="53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6" Sep 10 21:10:59.503604 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:10:59.503579 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6\": container with ID starting with 53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6 not found: ID does not exist" containerID="53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6" Sep 10 21:10:59.503714 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.503653 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6"} err="failed to get container status \"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6\": rpc error: code = NotFound desc = could not find container \"53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6\": container with ID starting with 53888c8250022b69dc94e08bf648ee4261064440bba083b9b63be39cc2fe4df6 not found: ID does not exist" Sep 10 21:10:59.573999 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.573972 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574014 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574040 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m95ng\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574252 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574252 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574174 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574252 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574202 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574252 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574240 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574432 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574281 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:10:59.574432 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574292 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data\") pod \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\" (UID: \"ce796f61-8e9d-4e04-97c7-6816c0c8a93c\") " Sep 10 21:10:59.574541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574423 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:10:59.574629 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574560 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:10:59.574699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574633 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:10:59.574699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574676 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:10:59.574936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574916 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.575006 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574939 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.575006 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574952 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.575006 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574965 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.575006 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.574978 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.576539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.576518 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:10:59.576645 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.576533 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng" (OuterVolumeSpecName: "kube-api-access-m95ng") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "kube-api-access-m95ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:10:59.576816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.576798 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:10:59.576997 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.576981 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce796f61-8e9d-4e04-97c7-6816c0c8a93c" (UID: "ce796f61-8e9d-4e04-97c7-6816c0c8a93c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:10:59.675513 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.675485 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.675513 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.675509 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95ng\" (UniqueName: \"kubernetes.io/projected/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-kube-api-access-m95ng\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.675691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.675522 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.675691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.675536 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce796f61-8e9d-4e04-97c7-6816c0c8a93c-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:10:59.827170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.827145 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:10:59.829141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:10:59.829122 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d9fd5e6e-openshift-default-86dfb5f864-lqmhk"] Sep 10 21:11:00.819366 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:00.819332 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" path="/var/lib/kubelet/pods/ce796f61-8e9d-4e04-97c7-6816c0c8a93c/volumes" Sep 10 21:11:20.597530 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.597491 2585 generic.go:356] "Generic (PLEG): container finished" podID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerID="ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605" exitCode=0 Sep 10 21:11:20.597987 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.597567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerDied","Data":"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605"} Sep 10 21:11:20.598041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.598031 2585 scope.go:117] "RemoveContainer" containerID="ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605" Sep 10 21:11:20.599132 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.599102 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:20.599232 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.599195 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:20.599296 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:20.599246 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:21.604501 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:21.604471 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerStarted","Data":"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2"} Sep 10 21:11:22.472256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.472219 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:22.472467 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.472402 2585 patch_prober.go:28] interesting pod/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 10 21:11:22.472467 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.472454 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 10 21:11:22.907296 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907262 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7"] Sep 10 21:11:22.907702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907687 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.907748 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907702 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.907748 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907726 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.907748 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907732 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.907841 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907807 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.907841 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.907817 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce796f61-8e9d-4e04-97c7-6816c0c8a93c" containerName="istio-proxy" Sep 10 21:11:22.910958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.910941 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:22.913378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.913354 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:11:22.913501 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.913354 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:11:22.920386 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:22.920362 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7"] Sep 10 21:11:23.000833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.000798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7\" (UID: \"9ce45799-1a34-4b8b-ac4b-90035533fe8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:23.102469 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.102380 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7\" (UID: \"9ce45799-1a34-4b8b-ac4b-90035533fe8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:23.103245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.103223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7\" (UID: \"9ce45799-1a34-4b8b-ac4b-90035533fe8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:23.221938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.221857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:23.472827 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.472750 2585 patch_prober.go:28] interesting pod/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 10 21:11:23.472827 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.472805 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 10 21:11:23.521284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.521254 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7"] Sep 10 21:11:23.523067 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:11:23.523036 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce45799_1a34_4b8b_ac4b_90035533fe8a.slice/crio-b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf WatchSource:0}: Error finding container b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf: Status 404 returned error can't find the container with id b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf Sep 10 21:11:23.621829 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:23.621792 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" event={"ID":"9ce45799-1a34-4b8b-ac4b-90035533fe8a","Type":"ContainerStarted","Data":"b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf"} Sep 10 21:11:24.473066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:24.473039 2585 patch_prober.go:28] interesting pod/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 10 21:11:24.473350 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:24.473103 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 10 21:11:24.629019 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:24.628983 2585 generic.go:356] "Generic (PLEG): container finished" podID="9ce45799-1a34-4b8b-ac4b-90035533fe8a" containerID="96329bb145bfb172f54c2507efc339a246e9b1b0b8760a44802aa99c454603b0" exitCode=0 Sep 10 21:11:24.629200 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:24.629074 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" event={"ID":"9ce45799-1a34-4b8b-ac4b-90035533fe8a","Type":"ContainerDied","Data":"96329bb145bfb172f54c2507efc339a246e9b1b0b8760a44802aa99c454603b0"} Sep 10 21:11:25.472306 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:25.472271 2585 patch_prober.go:28] interesting pod/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 10 21:11:25.472505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:25.472328 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 10 21:11:25.804475 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:25.804449 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:25.930056 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:25.930022 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca\") pod \"9ce45799-1a34-4b8b-ac4b-90035533fe8a\" (UID: \"9ce45799-1a34-4b8b-ac4b-90035533fe8a\") " Sep 10 21:11:25.930381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:25.930356 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ce45799-1a34-4b8b-ac4b-90035533fe8a" (UID: "9ce45799-1a34-4b8b-ac4b-90035533fe8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:11:26.031582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.031561 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce45799-1a34-4b8b-ac4b-90035533fe8a-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:26.473103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.473008 2585 patch_prober.go:28] interesting pod/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 10 21:11:26.473103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.473070 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 10 21:11:26.640449 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.640420 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" Sep 10 21:11:26.640635 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.640417 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7" event={"ID":"9ce45799-1a34-4b8b-ac4b-90035533fe8a","Type":"ContainerDied","Data":"b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf"} Sep 10 21:11:26.640635 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:26.640544 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77cb5af3bdf8f159769b2d7a95e3861776d00997e375670d6b801e99e0108cf" Sep 10 21:11:27.470897 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:27.470855 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:27.514763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:27.514732 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:27.652696 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:27.652668 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:30.257625 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:30.257577 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7"] Sep 10 21:11:30.260440 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:30.260412 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbddbd71-nclw7"] Sep 10 21:11:30.392931 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:30.392901 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:11:30.393143 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:30.393118 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" containerID="cri-o://a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2" gracePeriod=30 Sep 10 21:11:30.817967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:30.817935 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce45799-1a34-4b8b-ac4b-90035533fe8a" path="/var/lib/kubelet/pods/9ce45799-1a34-4b8b-ac4b-90035533fe8a/volumes" Sep 10 21:11:31.206209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.206133 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:11:31.206537 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.206525 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce45799-1a34-4b8b-ac4b-90035533fe8a" containerName="revoke-keys" Sep 10 21:11:31.206581 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.206538 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce45799-1a34-4b8b-ac4b-90035533fe8a" containerName="revoke-keys" Sep 10 21:11:31.206666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.206631 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce45799-1a34-4b8b-ac4b-90035533fe8a" containerName="revoke-keys" Sep 10 21:11:31.209985 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.209964 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.213702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.213682 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e95f46e9-openshift-default-dockercfg-d8thj\"" Sep 10 21:11:31.226188 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.226166 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:11:31.280359 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280396 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280419 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280452 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280561 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280652 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.280708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.280700 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lrn\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381657 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lrn\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381768 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.381915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382204 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382204 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.381979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382204 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382365 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382266 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382365 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382475 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382669 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.382783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.382707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.384419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.384400 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.384517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.384452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.391418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.391398 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.391562 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.391545 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lrn\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn\") pod \"e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.520000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.519912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:31.820566 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:11:31.820539 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9591480d_bceb_4a6d_9bfb_7ac76dcea094.slice/crio-c34e87c4ca4ec1c671ec748bf575af319f7f26fe51ce88db6bf93c2e5d1473e1 WatchSource:0}: Error finding container c34e87c4ca4ec1c671ec748bf575af319f7f26fe51ce88db6bf93c2e5d1473e1: Status 404 returned error can't find the container with id c34e87c4ca4ec1c671ec748bf575af319f7f26fe51ce88db6bf93c2e5d1473e1 Sep 10 21:11:31.821110 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.821019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:11:31.822904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.822870 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:31.823000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.822936 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:31.823000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:31.822984 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:32.672744 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:32.672709 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerStarted","Data":"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832"} Sep 10 21:11:32.673099 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:32.672761 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerStarted","Data":"c34e87c4ca4ec1c671ec748bf575af319f7f26fe51ce88db6bf93c2e5d1473e1"} Sep 10 21:11:32.701018 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:32.700967 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podStartSLOduration=1.700952 podStartE2EDuration="1.700952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:32.699450813 +0000 UTC m=+776.407745198" watchObservedRunningTime="2026-09-10 21:11:32.700952 +0000 UTC m=+776.409246386" Sep 10 21:11:33.520877 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:33.520841 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:33.525359 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:33.525334 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:33.678533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:33.678499 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:33.679275 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:33.679256 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:35.655436 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.655362 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:35.691025 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.690993 2585 generic.go:356] "Generic (PLEG): container finished" podID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerID="a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2" exitCode=0 Sep 10 21:11:35.691178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.691044 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" Sep 10 21:11:35.691178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.691071 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerDied","Data":"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2"} Sep 10 21:11:35.691178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.691150 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64" event={"ID":"149ce55f-cd0a-4fae-a090-d71c50a8ac34","Type":"ContainerDied","Data":"5c685d62c94021cd16c47c267c5fe9fa0ff672dc5f661bf1cfcbd8bd528f8af2"} Sep 10 21:11:35.691178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.691176 2585 scope.go:117] "RemoveContainer" containerID="a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2" Sep 10 21:11:35.699758 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.699734 2585 scope.go:117] "RemoveContainer" containerID="ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605" Sep 10 21:11:35.710331 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.710309 2585 scope.go:117] "RemoveContainer" containerID="a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2" Sep 10 21:11:35.710701 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:11:35.710673 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2\": container with ID starting with a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2 not found: ID does not exist" containerID="a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2" Sep 10 21:11:35.710780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.710714 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2"} err="failed to get container status \"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2\": rpc error: code = NotFound desc = could not find container \"a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2\": container with ID starting with a9153406a7692cacedd82a336193375cc097cd49fd92f74b81f7191d58d506c2 not found: ID does not exist" Sep 10 21:11:35.710780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.710740 2585 scope.go:117] "RemoveContainer" containerID="ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605" Sep 10 21:11:35.711117 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:11:35.711094 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605\": container with ID starting with ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605 not found: ID does not exist" containerID="ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605" Sep 10 21:11:35.711229 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.711121 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605"} err="failed to get container status \"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605\": rpc error: code = NotFound desc = could not find container \"ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605\": container with ID starting with ed669b82e74ef0c8a694e37d86952ac040c1d4f39cdcb48871d41e1b6ac78605 not found: ID does not exist" Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.725843 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.725913 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpk2g\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.725967 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726036 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726074 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726100 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726138 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726184 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726215 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy\") pod \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\" (UID: \"149ce55f-cd0a-4fae-a090-d71c50a8ac34\") " Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726513 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.726598 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.727070 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data" (OuterVolumeSpecName: "istio-data") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.727374 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:11:35.727526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.727471 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:35.728500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.727711 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:35.729534 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.729491 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:35.730724 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.730686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token" (OuterVolumeSpecName: "istio-token") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:11:35.731566 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.731538 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g" (OuterVolumeSpecName: "kube-api-access-zpk2g") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "kube-api-access-zpk2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:11:35.731688 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.731563 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "149ce55f-cd0a-4fae-a090-d71c50a8ac34" (UID: "149ce55f-cd0a-4fae-a090-d71c50a8ac34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827811 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpk2g\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-kube-api-access-zpk2g\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827845 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827859 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827873 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827886 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827899 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827910 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:35.828076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:35.827924 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149ce55f-cd0a-4fae-a090-d71c50a8ac34-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:11:36.022280 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:36.022246 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:11:36.027431 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:36.027405 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bbddbd71-gw-openshift-default-6888544fd4-lst64"] Sep 10 21:11:36.818284 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:36.818257 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" path="/var/lib/kubelet/pods/149ce55f-cd0a-4fae-a090-d71c50a8ac34/volumes" Sep 10 21:11:52.783409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.783376 2585 generic.go:356] "Generic (PLEG): container finished" podID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerID="6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832" exitCode=0 Sep 10 21:11:52.783805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.783429 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerDied","Data":"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832"} Sep 10 21:11:52.783848 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.783832 2585 scope.go:117] "RemoveContainer" containerID="6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832" Sep 10 21:11:52.784917 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.784887 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:52.785014 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.784953 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:52.785014 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:52.784982 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:11:53.800194 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:53.800154 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerStarted","Data":"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f"} Sep 10 21:11:54.520219 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:54.520182 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:54.520404 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:54.520374 2585 patch_prober.go:28] interesting pod/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 10 21:11:54.520450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:54.520421 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 10 21:11:55.520260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:55.520225 2585 patch_prober.go:28] interesting pod/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 10 21:11:55.520673 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:55.520280 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 10 21:11:56.520559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:56.520524 2585 patch_prober.go:28] interesting pod/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 10 21:11:56.520968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:56.520584 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 10 21:11:57.520916 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:57.520873 2585 patch_prober.go:28] interesting pod/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 10 21:11:57.521305 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:57.520952 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 10 21:11:58.520781 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:58.520738 2585 patch_prober.go:28] interesting pod/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 10 21:11:58.520983 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:58.520804 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 10 21:11:59.524642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.524550 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:59.525011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.524936 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:59.525646 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.525629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:11:59.771800 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.771770 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc"] Sep 10 21:11:59.772193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.772172 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:11:59.772193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.772189 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:11:59.772306 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.772260 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:11:59.772306 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.772269 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:11:59.775499 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.775457 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:11:59.778057 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.778037 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:11:59.778173 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.778036 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:11:59.786381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.786359 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc"] Sep 10 21:11:59.872246 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.872214 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc\" (UID: \"fda02aad-1be6-4fed-948d-e7360845908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:11:59.972844 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.972807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc\" (UID: \"fda02aad-1be6-4fed-948d-e7360845908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:11:59.973422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:11:59.973403 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc\" (UID: \"fda02aad-1be6-4fed-948d-e7360845908a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:12:00.086691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:00.086570 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:12:00.358768 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:00.358713 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc"] Sep 10 21:12:00.361052 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:00.361026 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda02aad_1be6_4fed_948d_e7360845908a.slice/crio-e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2 WatchSource:0}: Error finding container e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2: Status 404 returned error can't find the container with id e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2 Sep 10 21:12:00.837985 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:00.837952 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" event={"ID":"fda02aad-1be6-4fed-948d-e7360845908a","Type":"ContainerStarted","Data":"bc0638f70be727f6403e3a3951ffaeef3410849cfadc16ffdfa321581138e407"} Sep 10 21:12:00.838354 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:00.838002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" event={"ID":"fda02aad-1be6-4fed-948d-e7360845908a","Type":"ContainerStarted","Data":"e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2"} Sep 10 21:12:01.844191 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:01.844162 2585 generic.go:356] "Generic (PLEG): container finished" podID="fda02aad-1be6-4fed-948d-e7360845908a" containerID="bc0638f70be727f6403e3a3951ffaeef3410849cfadc16ffdfa321581138e407" exitCode=0 Sep 10 21:12:01.844557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:01.844254 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" event={"ID":"fda02aad-1be6-4fed-948d-e7360845908a","Type":"ContainerDied","Data":"bc0638f70be727f6403e3a3951ffaeef3410849cfadc16ffdfa321581138e407"} Sep 10 21:12:02.979886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:02.979861 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:12:03.100212 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.100144 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca\") pod \"fda02aad-1be6-4fed-948d-e7360845908a\" (UID: \"fda02aad-1be6-4fed-948d-e7360845908a\") " Sep 10 21:12:03.100521 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.100498 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fda02aad-1be6-4fed-948d-e7360845908a" (UID: "fda02aad-1be6-4fed-948d-e7360845908a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:03.201651 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.201602 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda02aad-1be6-4fed-948d-e7360845908a-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:03.855165 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.855132 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" Sep 10 21:12:03.855317 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.855177 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc" event={"ID":"fda02aad-1be6-4fed-948d-e7360845908a","Type":"ContainerDied","Data":"e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2"} Sep 10 21:12:03.855317 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:03.855220 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a395789487e16c49ee693c62ecc10448d148a806762431cdf0aae22c1075b2" Sep 10 21:12:06.418991 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:06.418956 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc"] Sep 10 21:12:06.421962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:06.421937 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e95f46e9-wsmkc"] Sep 10 21:12:06.817686 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:06.817654 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda02aad-1be6-4fed-948d-e7360845908a" path="/var/lib/kubelet/pods/fda02aad-1be6-4fed-948d-e7360845908a/volumes" Sep 10 21:12:07.568785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:07.568751 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:12:07.569168 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:07.568945 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" containerID="cri-o://0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f" gracePeriod=30 Sep 10 21:12:08.259053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259023 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:08.259426 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259414 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:12:08.259482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259428 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="149ce55f-cd0a-4fae-a090-d71c50a8ac34" containerName="istio-proxy" Sep 10 21:12:08.259482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259436 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda02aad-1be6-4fed-948d-e7360845908a" containerName="revoke-keys" Sep 10 21:12:08.259482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259442 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda02aad-1be6-4fed-948d-e7360845908a" containerName="revoke-keys" Sep 10 21:12:08.259591 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.259522 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda02aad-1be6-4fed-948d-e7360845908a" containerName="revoke-keys" Sep 10 21:12:08.262747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.262728 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.267042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.267022 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0d892491-openshift-default-dockercfg-bpr2p\"" Sep 10 21:12:08.279162 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.279140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:08.361347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361408 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4kw\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361516 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361636 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361688 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.361780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.361736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462368 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462368 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4kw\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462606 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.462912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.463072 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.462962 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.463072 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.463015 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.463072 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.463051 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.463414 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.463393 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.464844 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.464825 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.464933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.464915 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.472256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.472234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4kw\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.472352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.472269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token\") pod \"e2e-derive-0d892491-openshift-default-6b85768f6-svzpr\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.572630 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.572574 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:08.852462 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.852439 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:08.853766 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:08.853741 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2b2e8e_5acb_4d24_9d1c_1228901eaa0e.slice/crio-53ace38139a35ff50b0ce52856ace82496f2601d97a7c5ea08df6ba7d61c6f50 WatchSource:0}: Error finding container 53ace38139a35ff50b0ce52856ace82496f2601d97a7c5ea08df6ba7d61c6f50: Status 404 returned error can't find the container with id 53ace38139a35ff50b0ce52856ace82496f2601d97a7c5ea08df6ba7d61c6f50 Sep 10 21:12:08.858158 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.856596 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:08.858158 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.856765 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:08.858158 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.856848 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:08.881338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:08.881310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerStarted","Data":"53ace38139a35ff50b0ce52856ace82496f2601d97a7c5ea08df6ba7d61c6f50"} Sep 10 21:12:09.888546 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:09.888511 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerStarted","Data":"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413"} Sep 10 21:12:10.572856 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:10.572820 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:10.577934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:10.577909 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:10.605776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:10.605713 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podStartSLOduration=2.605697052 podStartE2EDuration="2.605697052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:09.916853102 +0000 UTC m=+813.625147487" watchObservedRunningTime="2026-09-10 21:12:10.605697052 +0000 UTC m=+814.313991671" Sep 10 21:12:10.894305 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:10.894229 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:10.895337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:10.895315 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:12.823212 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.823187 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:12:12.905103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.905024 2585 generic.go:356] "Generic (PLEG): container finished" podID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerID="0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f" exitCode=0 Sep 10 21:12:12.905103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.905074 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" Sep 10 21:12:12.905311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.905107 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerDied","Data":"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f"} Sep 10 21:12:12.905311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.905157 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv" event={"ID":"9591480d-bceb-4a6d-9bfb-7ac76dcea094","Type":"ContainerDied","Data":"c34e87c4ca4ec1c671ec748bf575af319f7f26fe51ce88db6bf93c2e5d1473e1"} Sep 10 21:12:12.905311 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.905178 2585 scope.go:117] "RemoveContainer" containerID="0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f" Sep 10 21:12:12.908425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908406 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908525 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908454 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908525 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908485 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908525 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908516 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908562 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908586 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908728 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:12.908919 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908744 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908919 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908773 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.908919 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908825 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69lrn\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn\") pod \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\" (UID: \"9591480d-bceb-4a6d-9bfb-7ac76dcea094\") " Sep 10 21:12:12.909059 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.908820 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:12.909224 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.909199 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data" (OuterVolumeSpecName: "istio-data") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:12.909280 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.909262 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:12.909336 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.909281 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:12.909625 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.909578 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:12.909729 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.909647 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:12.910981 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.910954 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:12.911384 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.911356 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn" (OuterVolumeSpecName: "kube-api-access-69lrn") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "kube-api-access-69lrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:12.911466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.911396 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token" (OuterVolumeSpecName: "istio-token") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:12.911584 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.911570 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9591480d-bceb-4a6d-9bfb-7ac76dcea094" (UID: "9591480d-bceb-4a6d-9bfb-7ac76dcea094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:12:12.914809 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.914793 2585 scope.go:117] "RemoveContainer" containerID="6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832" Sep 10 21:12:12.929775 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.929754 2585 scope.go:117] "RemoveContainer" containerID="0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f" Sep 10 21:12:12.930027 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:12.930007 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f\": container with ID starting with 0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f not found: ID does not exist" containerID="0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f" Sep 10 21:12:12.930077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.930042 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f"} err="failed to get container status \"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f\": rpc error: code = NotFound desc = could not find container \"0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f\": container with ID starting with 0d7f114c00d6ff19e5d0f9674c9b3a9f5879292df1230e921e0923bde800db5f not found: ID does not exist" Sep 10 21:12:12.930077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.930059 2585 scope.go:117] "RemoveContainer" containerID="6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832" Sep 10 21:12:12.930327 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:12.930307 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832\": container with ID starting with 6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832 not found: ID does not exist" containerID="6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832" Sep 10 21:12:12.930393 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:12.930330 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832"} err="failed to get container status \"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832\": rpc error: code = NotFound desc = could not find container \"6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832\": container with ID starting with 6e9ece00f28fed8866fe279ac1514ba062fec8319b33d062397d6e2be147c832 not found: ID does not exist" Sep 10 21:12:13.009833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009806 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.009833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009827 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.009833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009837 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.010053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009849 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.010053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009862 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.010053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009872 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9591480d-bceb-4a6d-9bfb-7ac76dcea094-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.010053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.009880 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69lrn\" (UniqueName: \"kubernetes.io/projected/9591480d-bceb-4a6d-9bfb-7ac76dcea094-kube-api-access-69lrn\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:13.236450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.236421 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:12:13.240763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:13.240734 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e95f46e9-openshift-default-78988fdcd6-6cczv"] Sep 10 21:12:14.818296 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:14.818260 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" path="/var/lib/kubelet/pods/9591480d-bceb-4a6d-9bfb-7ac76dcea094/volumes" Sep 10 21:12:18.618712 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.618682 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:18.619085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619069 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.619085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619080 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.619161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619101 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.619161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619107 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.619223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619161 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.619223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.619170 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9591480d-bceb-4a6d-9bfb-7ac76dcea094" containerName="istio-proxy" Sep 10 21:12:18.623533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.623516 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.627106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.627084 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f96a7a\"/\"kube-root-ca.crt\"" Sep 10 21:12:18.627106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.627106 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f96a7a\"/\"default-dockercfg-dvcjb\"" Sep 10 21:12:18.627250 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.627106 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f96a7a\"/\"openshift-service-ca.crt\"" Sep 10 21:12:18.627250 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.627088 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f96a7a\"/\"autores-model-f96a7a-kserve-self-signed-certs\"" Sep 10 21:12:18.634640 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.633092 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:18.759485 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759448 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.759701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759498 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.759701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.759701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.759701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.759904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.759769 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dgh\" (UniqueName: \"kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860410 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860376 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dgh\" (UniqueName: \"kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860545 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860647 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860701 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.860774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.860714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.861059 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.861033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.861182 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.861093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.861182 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.861055 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.862799 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.862771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.863033 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.863016 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.870994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.870926 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dgh\" (UniqueName: \"kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh\") pod \"autores-model-f96a7a-kserve-79cf67899f-qq22w\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:18.939821 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:18.939791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:19.237225 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:19.237200 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:19.238806 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:19.238766 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc72fbff0_ee7e_48e8_95a9_fbc621c79b27.slice/crio-059321919a05e228df596a91b1f3bd9ec2aae6cf9c31c0678e6e8447a452d5cf WatchSource:0}: Error finding container 059321919a05e228df596a91b1f3bd9ec2aae6cf9c31c0678e6e8447a452d5cf: Status 404 returned error can't find the container with id 059321919a05e228df596a91b1f3bd9ec2aae6cf9c31c0678e6e8447a452d5cf Sep 10 21:12:19.944540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:19.944503 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" event={"ID":"c72fbff0-ee7e-48e8-95a9-fbc621c79b27","Type":"ContainerStarted","Data":"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf"} Sep 10 21:12:19.944540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:19.944548 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" event={"ID":"c72fbff0-ee7e-48e8-95a9-fbc621c79b27","Type":"ContainerStarted","Data":"059321919a05e228df596a91b1f3bd9ec2aae6cf9c31c0678e6e8447a452d5cf"} Sep 10 21:12:19.962540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:19.962491 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" podStartSLOduration=1.9624765910000002 podStartE2EDuration="1.962476591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:19.961412646 +0000 UTC m=+823.669707032" watchObservedRunningTime="2026-09-10 21:12:19.962476591 +0000 UTC m=+823.670770976" Sep 10 21:12:28.939910 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:28.939868 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:28.940398 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:28.939926 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:28.953039 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:28.953014 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:29.000983 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:29.000954 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:29.999795 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:29.999705 2585 generic.go:356] "Generic (PLEG): container finished" podID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerID="3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413" exitCode=0 Sep 10 21:12:30.001321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:29.999958 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerDied","Data":"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413"} Sep 10 21:12:30.001966 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:30.001949 2585 scope.go:117] "RemoveContainer" containerID="3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413" Sep 10 21:12:30.003436 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:30.003407 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:30.003653 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:30.003630 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:30.003767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:30.003744 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:31.008080 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.008046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerStarted","Data":"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa"} Sep 10 21:12:31.342285 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.342249 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:31.464716 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.464686 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw"] Sep 10 21:12:31.468270 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.468254 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:31.470816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.470784 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:12:31.470816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.470801 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:12:31.481124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.478273 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw"] Sep 10 21:12:31.494669 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.494494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f96a7a-662zw\" (UID: \"b57808b2-e2b4-4367-b4d4-1dc3d5f26018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:31.573194 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.573162 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:31.573364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.573338 2585 patch_prober.go:28] interesting pod/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 21:12:31.573413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.573387 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 21:12:31.597055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.596000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f96a7a-662zw\" (UID: \"b57808b2-e2b4-4367-b4d4-1dc3d5f26018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:31.597055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.596954 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f96a7a-662zw\" (UID: \"b57808b2-e2b4-4367-b4d4-1dc3d5f26018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:31.784399 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:31.784360 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:32.058958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:32.058933 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw"] Sep 10 21:12:32.060990 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:32.060967 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57808b2_e2b4_4367_b4d4_1dc3d5f26018.slice/crio-1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730 WatchSource:0}: Error finding container 1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730: Status 404 returned error can't find the container with id 1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730 Sep 10 21:12:32.574014 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:32.573969 2585 patch_prober.go:28] interesting pod/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 21:12:32.574215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:32.574039 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 21:12:33.020179 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:33.020149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" event={"ID":"b57808b2-e2b4-4367-b4d4-1dc3d5f26018","Type":"ContainerStarted","Data":"6d11610eaed61d86a656a96c8ffd7b91a65eb004aef7bc5a5d469a2b3ff7b48e"} Sep 10 21:12:33.020294 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:33.020204 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" event={"ID":"b57808b2-e2b4-4367-b4d4-1dc3d5f26018","Type":"ContainerStarted","Data":"1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730"} Sep 10 21:12:33.038897 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:33.038851 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" podStartSLOduration=2.038835981 podStartE2EDuration="2.038835981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:33.038418223 +0000 UTC m=+836.746712610" watchObservedRunningTime="2026-09-10 21:12:33.038835981 +0000 UTC m=+836.747130367" Sep 10 21:12:33.573337 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:33.573309 2585 patch_prober.go:28] interesting pod/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 21:12:33.573764 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:33.573360 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 21:12:34.026915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:34.026883 2585 generic.go:356] "Generic (PLEG): container finished" podID="b57808b2-e2b4-4367-b4d4-1dc3d5f26018" containerID="6d11610eaed61d86a656a96c8ffd7b91a65eb004aef7bc5a5d469a2b3ff7b48e" exitCode=0 Sep 10 21:12:34.027081 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:34.026969 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" event={"ID":"b57808b2-e2b4-4367-b4d4-1dc3d5f26018","Type":"ContainerDied","Data":"6d11610eaed61d86a656a96c8ffd7b91a65eb004aef7bc5a5d469a2b3ff7b48e"} Sep 10 21:12:34.573470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:34.573431 2585 patch_prober.go:28] interesting pod/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 21:12:34.573972 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:34.573505 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 21:12:35.227007 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.226984 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:35.235355 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.235332 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca\") pod \"b57808b2-e2b4-4367-b4d4-1dc3d5f26018\" (UID: \"b57808b2-e2b4-4367-b4d4-1dc3d5f26018\") " Sep 10 21:12:35.235678 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.235656 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b57808b2-e2b4-4367-b4d4-1dc3d5f26018" (UID: "b57808b2-e2b4-4367-b4d4-1dc3d5f26018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:35.336184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.336103 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57808b2-e2b4-4367-b4d4-1dc3d5f26018-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:35.582805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.582763 2585 patch_prober.go:28] interesting pod/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 21:12:35.583404 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:35.583367 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 21:12:36.039469 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.039436 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" Sep 10 21:12:36.039469 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.039463 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw" event={"ID":"b57808b2-e2b4-4367-b4d4-1dc3d5f26018","Type":"ContainerDied","Data":"1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730"} Sep 10 21:12:36.039714 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.039508 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dcfd7c48ac970392d1d78a825984fc03b3b172636feaf51420a1d9596828730" Sep 10 21:12:36.079927 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.079888 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg"] Sep 10 21:12:36.080538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.080511 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57808b2-e2b4-4367-b4d4-1dc3d5f26018" containerName="revoke-keys" Sep 10 21:12:36.080538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.080539 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57808b2-e2b4-4367-b4d4-1dc3d5f26018" containerName="revoke-keys" Sep 10 21:12:36.080742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.080720 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57808b2-e2b4-4367-b4d4-1dc3d5f26018" containerName="revoke-keys" Sep 10 21:12:36.088410 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.088376 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:36.091848 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.091818 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg"] Sep 10 21:12:36.092921 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.092895 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:12:36.093038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.092984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:12:36.143776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.143746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d892491-mx2fg\" (UID: \"a6247afc-eb38-49d7-9694-d09428b0428e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:36.245271 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.245238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d892491-mx2fg\" (UID: \"a6247afc-eb38-49d7-9694-d09428b0428e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:36.245868 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.245849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d892491-mx2fg\" (UID: \"a6247afc-eb38-49d7-9694-d09428b0428e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:36.409791 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.409685 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:36.626530 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.626498 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:36.627107 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.627058 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:36.627914 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.627884 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:36.692682 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:36.692656 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg"] Sep 10 21:12:36.694625 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:36.694584 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6247afc_eb38_49d7_9694_d09428b0428e.slice/crio-d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228 WatchSource:0}: Error finding container d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228: Status 404 returned error can't find the container with id d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228 Sep 10 21:12:37.046223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:37.046189 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" event={"ID":"a6247afc-eb38-49d7-9694-d09428b0428e","Type":"ContainerStarted","Data":"df4236aaf132bd902ad80348b99586e38f36eec03ed5bdb3b8204e57602b5ade"} Sep 10 21:12:37.046357 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:37.046246 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" event={"ID":"a6247afc-eb38-49d7-9694-d09428b0428e","Type":"ContainerStarted","Data":"d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228"} Sep 10 21:12:37.064010 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:37.063966 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" podStartSLOduration=1.063951283 podStartE2EDuration="1.063951283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:37.061925513 +0000 UTC m=+840.770219899" watchObservedRunningTime="2026-09-10 21:12:37.063951283 +0000 UTC m=+840.772245660" Sep 10 21:12:38.052498 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:38.052463 2585 generic.go:356] "Generic (PLEG): container finished" podID="a6247afc-eb38-49d7-9694-d09428b0428e" containerID="df4236aaf132bd902ad80348b99586e38f36eec03ed5bdb3b8204e57602b5ade" exitCode=0 Sep 10 21:12:38.052906 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:38.052558 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" event={"ID":"a6247afc-eb38-49d7-9694-d09428b0428e","Type":"ContainerDied","Data":"df4236aaf132bd902ad80348b99586e38f36eec03ed5bdb3b8204e57602b5ade"} Sep 10 21:12:39.183193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:39.183167 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:39.273326 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:39.273297 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca\") pod \"a6247afc-eb38-49d7-9694-d09428b0428e\" (UID: \"a6247afc-eb38-49d7-9694-d09428b0428e\") " Sep 10 21:12:39.273694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:39.273670 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6247afc-eb38-49d7-9694-d09428b0428e" (UID: "a6247afc-eb38-49d7-9694-d09428b0428e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:39.374215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:39.374152 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6247afc-eb38-49d7-9694-d09428b0428e-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:40.064190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:40.064163 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" Sep 10 21:12:40.064355 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:40.064209 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg" event={"ID":"a6247afc-eb38-49d7-9694-d09428b0428e","Type":"ContainerDied","Data":"d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228"} Sep 10 21:12:40.064355 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:40.064259 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ff73eaee49554139bab3fef3da1fc1a985fb591ba742c3dbc882464e48b228" Sep 10 21:12:42.465777 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:42.465743 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw"] Sep 10 21:12:42.468574 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:42.468550 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f96a7a-662zw"] Sep 10 21:12:42.818279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:42.818249 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57808b2-e2b4-4367-b4d4-1dc3d5f26018" path="/var/lib/kubelet/pods/b57808b2-e2b4-4367-b4d4-1dc3d5f26018/volumes" Sep 10 21:12:43.551277 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:43.551246 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg"] Sep 10 21:12:43.556719 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:43.556694 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d892491-mx2fg"] Sep 10 21:12:43.684959 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:43.684919 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:43.685329 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:43.685297 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" containerID="cri-o://2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa" gracePeriod=30 Sep 10 21:12:44.818546 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:44.818514 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6247afc-eb38-49d7-9694-d09428b0428e" path="/var/lib/kubelet/pods/a6247afc-eb38-49d7-9694-d09428b0428e/volumes" Sep 10 21:12:46.362660 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.362621 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" podUID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" containerName="main" containerID="cri-o://5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf" gracePeriod=45 Sep 10 21:12:46.616096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.616035 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:46.642032 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642004 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642168 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642042 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642168 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642083 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dgh\" (UniqueName: \"kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642168 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642140 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642289 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642260 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642349 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache\") pod \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\" (UID: \"c72fbff0-ee7e-48e8-95a9-fbc621c79b27\") " Sep 10 21:12:46.642418 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642357 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:46.642816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642699 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache" (OuterVolumeSpecName: "model-cache") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:46.642816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home" (OuterVolumeSpecName: "home") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:46.642975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642888 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tmp-dir\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:46.642975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642906 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-home\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:46.642975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.642918 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-model-cache\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:46.644304 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.644279 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm" (OuterVolumeSpecName: "dshm") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:46.644561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.644534 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:12:46.644659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.644582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh" (OuterVolumeSpecName: "kube-api-access-24dgh") pod "c72fbff0-ee7e-48e8-95a9-fbc621c79b27" (UID: "c72fbff0-ee7e-48e8-95a9-fbc621c79b27"). InnerVolumeSpecName "kube-api-access-24dgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:46.744439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.744410 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-dshm\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:46.744439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.744434 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-tls-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:46.744634 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:46.744446 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dgh\" (UniqueName: \"kubernetes.io/projected/c72fbff0-ee7e-48e8-95a9-fbc621c79b27-kube-api-access-24dgh\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:47.104265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.104231 2585 generic.go:356] "Generic (PLEG): container finished" podID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" containerID="5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf" exitCode=0 Sep 10 21:12:47.104424 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.104275 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" Sep 10 21:12:47.104424 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.104318 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" event={"ID":"c72fbff0-ee7e-48e8-95a9-fbc621c79b27","Type":"ContainerDied","Data":"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf"} Sep 10 21:12:47.104424 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.104370 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w" event={"ID":"c72fbff0-ee7e-48e8-95a9-fbc621c79b27","Type":"ContainerDied","Data":"059321919a05e228df596a91b1f3bd9ec2aae6cf9c31c0678e6e8447a452d5cf"} Sep 10 21:12:47.104424 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.104392 2585 scope.go:117] "RemoveContainer" containerID="5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf" Sep 10 21:12:47.112754 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.112735 2585 scope.go:117] "RemoveContainer" containerID="5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf" Sep 10 21:12:47.113055 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:47.113034 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf\": container with ID starting with 5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf not found: ID does not exist" containerID="5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf" Sep 10 21:12:47.113104 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.113064 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf"} err="failed to get container status \"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf\": rpc error: code = NotFound desc = could not find container \"5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf\": container with ID starting with 5b975c6f632376ba988fb46fc07d904b90faf20a72290ccf5aff76719db66ebf not found: ID does not exist" Sep 10 21:12:47.131894 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.131851 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:47.134156 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:47.134129 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f96a7a/autores-model-f96a7a-kserve-79cf67899f-qq22w"] Sep 10 21:12:48.818530 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:48.818499 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" path="/var/lib/kubelet/pods/c72fbff0-ee7e-48e8-95a9-fbc621c79b27/volumes" Sep 10 21:12:48.957852 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:48.957795 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:49.066775 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066742 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.066967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066802 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.066967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066839 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.066967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066882 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.066967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066909 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.066967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066933 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv4kw\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.067227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.066976 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.067227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067030 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.067227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067101 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:49.067227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067136 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy\") pod \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\" (UID: \"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e\") " Sep 10 21:12:49.067227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067164 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:49.067487 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067422 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:49.067666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067606 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:49.067666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067660 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.067816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067685 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.067816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067707 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.067816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.067765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:49.069352 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.069327 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:12:49.069543 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.069519 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:49.069658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.069639 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw" (OuterVolumeSpecName: "kube-api-access-zv4kw") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "kube-api-access-zv4kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:49.069736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.069721 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" (UID: "1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:49.089719 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.089688 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:12:49.090112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.089953 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" containerID="cri-o://9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1" gracePeriod=30 Sep 10 21:12:49.117419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.117392 2585 generic.go:356] "Generic (PLEG): container finished" podID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerID="2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa" exitCode=0 Sep 10 21:12:49.117561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.117445 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" Sep 10 21:12:49.117561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.117466 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerDied","Data":"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa"} Sep 10 21:12:49.117561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.117534 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr" event={"ID":"1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e","Type":"ContainerDied","Data":"53ace38139a35ff50b0ce52856ace82496f2601d97a7c5ea08df6ba7d61c6f50"} Sep 10 21:12:49.117773 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.117571 2585 scope.go:117] "RemoveContainer" containerID="2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa" Sep 10 21:12:49.128959 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.128937 2585 scope.go:117] "RemoveContainer" containerID="3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413" Sep 10 21:12:49.136798 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.136783 2585 scope.go:117] "RemoveContainer" containerID="2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa" Sep 10 21:12:49.137064 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:49.137047 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa\": container with ID starting with 2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa not found: ID does not exist" containerID="2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa" Sep 10 21:12:49.137108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.137072 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa"} err="failed to get container status \"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa\": rpc error: code = NotFound desc = could not find container \"2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa\": container with ID starting with 2cba5df57e1535ed1d773c7c94c55188172d2d4f6c5e6e61b823f6b58cf8c5aa not found: ID does not exist" Sep 10 21:12:49.137108 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.137089 2585 scope.go:117] "RemoveContainer" containerID="3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413" Sep 10 21:12:49.137309 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:49.137293 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413\": container with ID starting with 3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413 not found: ID does not exist" containerID="3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413" Sep 10 21:12:49.137353 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.137313 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413"} err="failed to get container status \"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413\": rpc error: code = NotFound desc = could not find container \"3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413\": container with ID starting with 3e4ca39b711dbeb5aa5e5eddc4b5a3e53f639b61656f65bc4d0038e58ac57413 not found: ID does not exist" Sep 10 21:12:49.164391 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.164364 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:49.168214 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168195 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4kw\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-kube-api-access-zv4kw\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.168214 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168215 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.168334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168226 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.168334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168234 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.168334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168242 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.168334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.168252 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:49.173750 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.173729 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0d892491-openshift-default-6b85768f6-svzpr"] Sep 10 21:12:49.822810 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.822782 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:12:49.823209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823191 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" containerName="main" Sep 10 21:12:49.823209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823207 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" containerName="main" Sep 10 21:12:49.823209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823215 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6247afc-eb38-49d7-9694-d09428b0428e" containerName="revoke-keys" Sep 10 21:12:49.823209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823220 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6247afc-eb38-49d7-9694-d09428b0428e" containerName="revoke-keys" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823230 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823239 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823249 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823259 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823341 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72fbff0-ee7e-48e8-95a9-fbc621c79b27" containerName="main" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823349 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:12:49.823439 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.823356 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6247afc-eb38-49d7-9694-d09428b0428e" containerName="revoke-keys" Sep 10 21:12:49.827860 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.827842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.830989 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.830971 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ea71f9-openshift-default-dockercfg-h9mz7\"" Sep 10 21:12:49.846163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.846138 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:12:49.873264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873241 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873274 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873347 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873491 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873408 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873491 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873427 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6s2\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873491 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873459 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873491 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.873663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.873498 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974597 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974683 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6s2\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.974831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975179 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974913 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975179 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.974945 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975649 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.975597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975766 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.975641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975766 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.975677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975766 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.975700 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.975929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.975831 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.977194 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.977174 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.977602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.977573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.994808 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.994788 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6s2\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:49.995896 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:49.995876 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token\") pod \"e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:50.138166 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.138095 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:50.415426 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.415403 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:12:50.417082 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:12:50.417054 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b253e4c_79f2_42a5_a239_41ef4514375b.slice/crio-458b020c13b5356297e2642378d65dc91a48669f728bf08da9b9273b757ac6ad WatchSource:0}: Error finding container 458b020c13b5356297e2642378d65dc91a48669f728bf08da9b9273b757ac6ad: Status 404 returned error can't find the container with id 458b020c13b5356297e2642378d65dc91a48669f728bf08da9b9273b757ac6ad Sep 10 21:12:50.419565 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.419535 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:50.419662 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.419597 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:50.419700 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.419661 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:12:50.818597 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:50.818568 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" path="/var/lib/kubelet/pods/1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e/volumes" Sep 10 21:12:51.130990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:51.130913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerStarted","Data":"99147a1e091b37b957dd5a7b7f4ca01041aca071b1f75103e1b7bd4cf4446e1c"} Sep 10 21:12:51.130990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:51.130968 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerStarted","Data":"458b020c13b5356297e2642378d65dc91a48669f728bf08da9b9273b757ac6ad"} Sep 10 21:12:51.138386 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:51.138363 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:51.143150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:51.143131 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:51.153677 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:51.153564 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podStartSLOduration=2.153549217 podStartE2EDuration="2.153549217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:51.152581641 +0000 UTC m=+854.860876027" watchObservedRunningTime="2026-09-10 21:12:51.153549217 +0000 UTC m=+854.861843603" Sep 10 21:12:52.137896 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:52.137862 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:52.138901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:52.138880 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:12:54.337228 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.337202 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:12:54.420729 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420660 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420729 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420718 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420744 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420774 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420845 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420891 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.420936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420917 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.421189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420970 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.421189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.420990 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:54.421189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421005 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mndd\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd\") pod \"1ab11ec1-26dc-4db2-97f4-77c85667517f\" (UID: \"1ab11ec1-26dc-4db2-97f4-77c85667517f\") " Sep 10 21:12:54.421189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421156 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:54.421189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421178 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:54.421466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421295 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:54.421466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421364 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:54.421466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421437 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.421466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421453 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab11ec1-26dc-4db2-97f4-77c85667517f-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.421466 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421465 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.421745 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.421474 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.422926 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.422905 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:54.423107 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.423087 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd" (OuterVolumeSpecName: "kube-api-access-6mndd") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "kube-api-access-6mndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:54.423354 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.423338 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:12:54.423399 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.423360 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ab11ec1-26dc-4db2-97f4-77c85667517f" (UID: "1ab11ec1-26dc-4db2-97f4-77c85667517f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:12:54.522297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.522265 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.522297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.522290 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab11ec1-26dc-4db2-97f4-77c85667517f-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.522297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.522301 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mndd\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-kube-api-access-6mndd\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.522524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.522309 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:54.522524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:54.522318 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab11ec1-26dc-4db2-97f4-77c85667517f-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:12:55.162674 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.162561 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerID="9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1" exitCode=0 Sep 10 21:12:55.162674 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.162638 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" Sep 10 21:12:55.162674 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.162658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerDied","Data":"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1"} Sep 10 21:12:55.162923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.162731 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp" event={"ID":"1ab11ec1-26dc-4db2-97f4-77c85667517f","Type":"ContainerDied","Data":"d81474d99cc8632d69425fb35e1bb958bf577f13d162f1be877189edc5aa8563"} Sep 10 21:12:55.162923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.162753 2585 scope.go:117] "RemoveContainer" containerID="9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1" Sep 10 21:12:55.172116 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.172099 2585 scope.go:117] "RemoveContainer" containerID="07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6" Sep 10 21:12:55.180047 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.180028 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:12:55.180445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.180427 2585 scope.go:117] "RemoveContainer" containerID="9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1" Sep 10 21:12:55.180693 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:55.180677 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1\": container with ID starting with 9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1 not found: ID does not exist" containerID="9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1" Sep 10 21:12:55.180767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.180700 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1"} err="failed to get container status \"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1\": rpc error: code = NotFound desc = could not find container \"9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1\": container with ID starting with 9dbd1026c4cd7928bf957c88b499e569373b30b01139d4e8a830e2a3c81a00e1 not found: ID does not exist" Sep 10 21:12:55.180767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.180714 2585 scope.go:117] "RemoveContainer" containerID="07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6" Sep 10 21:12:55.181003 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:12:55.180979 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6\": container with ID starting with 07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6 not found: ID does not exist" containerID="07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6" Sep 10 21:12:55.181053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.181012 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6"} err="failed to get container status \"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6\": rpc error: code = NotFound desc = could not find container \"07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6\": container with ID starting with 07385b201cd355c56f8fe10c204d0b97553c21534d05efc3ac6a00c25bf2deb6 not found: ID does not exist" Sep 10 21:12:55.183962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:55.183942 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f96a7a-openshift-default-747d475c7d-gx9bp"] Sep 10 21:12:56.818398 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:12:56.818364 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" path="/var/lib/kubelet/pods/1ab11ec1-26dc-4db2-97f4-77c85667517f/volumes" Sep 10 21:13:08.179757 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:08.179722 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 21:13:08.180222 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:08.179787 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 21:13:10.245559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.245525 2585 generic.go:356] "Generic (PLEG): container finished" podID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerID="99147a1e091b37b957dd5a7b7f4ca01041aca071b1f75103e1b7bd4cf4446e1c" exitCode=0 Sep 10 21:13:10.245987 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.245604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerDied","Data":"99147a1e091b37b957dd5a7b7f4ca01041aca071b1f75103e1b7bd4cf4446e1c"} Sep 10 21:13:10.246060 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.246046 2585 scope.go:117] "RemoveContainer" containerID="99147a1e091b37b957dd5a7b7f4ca01041aca071b1f75103e1b7bd4cf4446e1c" Sep 10 21:13:10.247113 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.247086 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:10.247215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.247165 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:10.247277 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:10.247213 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:11.252741 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:11.252707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerStarted","Data":"70a428b7d9eec6b672d378a3be6997265465231667b2b7466cd89c836abbfe5b"} Sep 10 21:13:12.139244 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:12.139205 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:12.139428 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:12.139410 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 21:13:12.139481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:12.139461 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 21:13:13.139420 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:13.139387 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 21:13:13.139806 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:13.139439 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 21:13:14.139021 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:14.138981 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 21:13:14.139202 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:14.139042 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 21:13:15.139068 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:15.139031 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 21:13:15.139521 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:15.139092 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 21:13:16.139339 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:16.139298 2585 patch_prober.go:28] interesting pod/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 21:13:16.139835 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:16.139361 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 21:13:17.142727 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:17.142694 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:17.143197 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:17.143028 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:17.143782 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:17.143765 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:30.730574 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.730497 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28"] Sep 10 21:13:30.731027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.730945 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:30.731027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.730962 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:30.731157 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.731045 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:30.731157 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.731060 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2b2e8e-5acb-4d24-9d1c-1228901eaa0e" containerName="istio-proxy" Sep 10 21:13:30.734414 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.734394 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:30.736765 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.736746 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:13:30.736874 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.736764 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:13:30.742139 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.742117 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28"] Sep 10 21:13:30.864226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.864196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea71f9-mct28\" (UID: \"b217b239-4ae1-4f69-b1af-acb8c883af09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:30.964837 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.964811 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea71f9-mct28\" (UID: \"b217b239-4ae1-4f69-b1af-acb8c883af09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:30.965396 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:30.965377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ea71f9-mct28\" (UID: \"b217b239-4ae1-4f69-b1af-acb8c883af09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:31.070356 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:31.070324 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:31.340093 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:31.340020 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28"] Sep 10 21:13:31.343221 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:13:31.343199 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb217b239_4ae1_4f69_b1af_acb8c883af09.slice/crio-ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403 WatchSource:0}: Error finding container ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403: Status 404 returned error can't find the container with id ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403 Sep 10 21:13:31.365031 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:31.365004 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" event={"ID":"b217b239-4ae1-4f69-b1af-acb8c883af09","Type":"ContainerStarted","Data":"ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403"} Sep 10 21:13:32.371567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:32.371481 2585 generic.go:356] "Generic (PLEG): container finished" podID="b217b239-4ae1-4f69-b1af-acb8c883af09" containerID="7bae7dc8e6498a829055728fa15dc99b7100da6f08331572eba9a695f40970b3" exitCode=0 Sep 10 21:13:32.371567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:32.371546 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" event={"ID":"b217b239-4ae1-4f69-b1af-acb8c883af09","Type":"ContainerDied","Data":"7bae7dc8e6498a829055728fa15dc99b7100da6f08331572eba9a695f40970b3"} Sep 10 21:13:33.508199 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:33.508179 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:33.590325 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:33.590295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca\") pod \"b217b239-4ae1-4f69-b1af-acb8c883af09\" (UID: \"b217b239-4ae1-4f69-b1af-acb8c883af09\") " Sep 10 21:13:33.590630 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:33.590588 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b217b239-4ae1-4f69-b1af-acb8c883af09" (UID: "b217b239-4ae1-4f69-b1af-acb8c883af09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:13:33.691549 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:33.691476 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b217b239-4ae1-4f69-b1af-acb8c883af09-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:34.385724 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:34.385690 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" event={"ID":"b217b239-4ae1-4f69-b1af-acb8c883af09","Type":"ContainerDied","Data":"ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403"} Sep 10 21:13:34.385907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:34.385739 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb9119afe82ebf6f6759a6b95f87fad3f37a2b2d28dcb5a7fa4b3e0ee2c8403" Sep 10 21:13:34.385907 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:34.385748 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28" Sep 10 21:13:36.808602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:36.808571 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:13:36.809996 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:36.809974 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:13:36.817881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:36.817859 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:13:36.819305 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:36.819281 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:13:41.700051 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:41.699899 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28"] Sep 10 21:13:41.701843 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:41.701820 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ea71f9-mct28"] Sep 10 21:13:42.817937 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:42.817907 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b217b239-4ae1-4f69-b1af-acb8c883af09" path="/var/lib/kubelet/pods/b217b239-4ae1-4f69-b1af-acb8c883af09/volumes" Sep 10 21:13:49.340239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:49.340204 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:13:49.340730 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:49.340402 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" containerID="cri-o://70a428b7d9eec6b672d378a3be6997265465231667b2b7466cd89c836abbfe5b" gracePeriod=30 Sep 10 21:13:50.060348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060318 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:13:50.060819 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060802 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:50.060904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060821 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:50.060904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060837 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b217b239-4ae1-4f69-b1af-acb8c883af09" containerName="revoke-keys" Sep 10 21:13:50.060904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060846 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b217b239-4ae1-4f69-b1af-acb8c883af09" containerName="revoke-keys" Sep 10 21:13:50.061024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060980 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b217b239-4ae1-4f69-b1af-acb8c883af09" containerName="revoke-keys" Sep 10 21:13:50.061024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.060997 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab11ec1-26dc-4db2-97f4-77c85667517f" containerName="istio-proxy" Sep 10 21:13:50.065722 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.065696 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.068197 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.068174 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-21de0f-openshift-default-dockercfg-2bwms\"" Sep 10 21:13:50.081143 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.081116 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:13:50.139627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139643 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139706 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139813 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5m46\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139948 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.139996 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.139978 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.140038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.140007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241064 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241064 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241068 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5m46\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241090 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241159 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241191 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241288 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241319 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241355 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241572 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.241887 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241862 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.242011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.242011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.241959 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.243642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.243601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.243874 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.243856 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.249488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.249462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.249650 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.249635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5m46\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46\") pod \"e2e-noten-21de0f-openshift-default-5d987c799b-crn5h\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.375485 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.375418 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:50.679279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.679205 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:13:50.687664 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:13:50.687638 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065ee6ba_6af3_4fe3_a677_f0f8fbfeab9d.slice/crio-6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a WatchSource:0}: Error finding container 6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a: Status 404 returned error can't find the container with id 6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a Sep 10 21:13:50.689911 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.689876 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:50.690027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.689945 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:50.690027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:50.689974 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:13:51.489185 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:51.489146 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" event={"ID":"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d","Type":"ContainerStarted","Data":"7b3a30500bfebb3886ba6e67fef890ca5b6077161c867a63271d846066ced6de"} Sep 10 21:13:51.489648 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:51.489215 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" event={"ID":"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d","Type":"ContainerStarted","Data":"6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a"} Sep 10 21:13:51.513397 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:51.513334 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" podStartSLOduration=1.513320737 podStartE2EDuration="1.513320737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:51.513034443 +0000 UTC m=+915.221328830" watchObservedRunningTime="2026-09-10 21:13:51.513320737 +0000 UTC m=+915.221615126" Sep 10 21:13:52.375814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:52.375784 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:52.380957 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:52.380931 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:52.495583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:52.495547 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:52.496524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:52.496504 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:13:54.509906 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.509875 2585 generic.go:356] "Generic (PLEG): container finished" podID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerID="70a428b7d9eec6b672d378a3be6997265465231667b2b7466cd89c836abbfe5b" exitCode=0 Sep 10 21:13:54.510273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.509944 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerDied","Data":"70a428b7d9eec6b672d378a3be6997265465231667b2b7466cd89c836abbfe5b"} Sep 10 21:13:54.510273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.510010 2585 scope.go:117] "RemoveContainer" containerID="99147a1e091b37b957dd5a7b7f4ca01041aca071b1f75103e1b7bd4cf4446e1c" Sep 10 21:13:54.587304 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.587277 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:54.683338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683255 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683325 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683344 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683367 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683401 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683446 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc6s2\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683485 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683509 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683566 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert\") pod \"6b253e4c-79f2-42a5-a239-41ef4514375b\" (UID: \"6b253e4c-79f2-42a5-a239-41ef4514375b\") " Sep 10 21:13:54.683961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683695 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:13:54.683961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683743 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:13:54.683961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683727 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:13:54.683961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.683813 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:13:54.684170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.684041 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:13:54.684170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.684047 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.684170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.684084 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.684170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.684098 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.684170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.684120 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.685496 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.685475 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:13:54.686000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.685976 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:13:54.686077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.685994 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2" (OuterVolumeSpecName: "kube-api-access-jc6s2") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "kube-api-access-jc6s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:13:54.686077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.686010 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b253e4c-79f2-42a5-a239-41ef4514375b" (UID: "6b253e4c-79f2-42a5-a239-41ef4514375b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:13:54.785340 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.785317 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6s2\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-kube-api-access-jc6s2\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.785340 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.785338 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.785470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.785348 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b253e4c-79f2-42a5-a239-41ef4514375b-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.785470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.785356 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:54.785470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:54.785363 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b253e4c-79f2-42a5-a239-41ef4514375b-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:13:55.516207 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:55.516173 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" event={"ID":"6b253e4c-79f2-42a5-a239-41ef4514375b","Type":"ContainerDied","Data":"458b020c13b5356297e2642378d65dc91a48669f728bf08da9b9273b757ac6ad"} Sep 10 21:13:55.516207 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:55.516202 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj" Sep 10 21:13:55.516709 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:55.516253 2585 scope.go:117] "RemoveContainer" containerID="70a428b7d9eec6b672d378a3be6997265465231667b2b7466cd89c836abbfe5b" Sep 10 21:13:55.543409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:55.543387 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:13:55.548978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:55.548933 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ea71f9-openshift-default-585cb6ddbd-h4shj"] Sep 10 21:13:56.817669 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:13:56.817640 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" path="/var/lib/kubelet/pods/6b253e4c-79f2-42a5-a239-41ef4514375b/volumes" Sep 10 21:14:01.329661 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:01.329606 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:14:01.330075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:01.329870 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" podUID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" containerName="istio-proxy" containerID="cri-o://7b3a30500bfebb3886ba6e67fef890ca5b6077161c867a63271d846066ced6de" gracePeriod=30 Sep 10 21:14:06.574270 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.574241 2585 generic.go:356] "Generic (PLEG): container finished" podID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" containerID="7b3a30500bfebb3886ba6e67fef890ca5b6077161c867a63271d846066ced6de" exitCode=0 Sep 10 21:14:06.574605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.574310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" event={"ID":"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d","Type":"ContainerDied","Data":"7b3a30500bfebb3886ba6e67fef890ca5b6077161c867a63271d846066ced6de"} Sep 10 21:14:06.574605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.574364 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" event={"ID":"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d","Type":"ContainerDied","Data":"6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a"} Sep 10 21:14:06.574605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.574380 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a91274035907481f577c23308bdb2c96ed1eda40ab055cd7f6ab21b17d9023a" Sep 10 21:14:06.582194 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.582177 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:14:06.692244 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692170 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692244 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692203 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692244 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692240 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692268 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692288 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692318 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5m46\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692401 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692517 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data\") pod \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\" (UID: \"065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d\") " Sep 10 21:14:06.692785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692501 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:06.692785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692575 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:06.692785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692680 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:14:06.692785 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692685 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692844 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692944 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692963 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692977 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.692990 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.693069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.693011 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.694398 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.694377 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:06.694720 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.694696 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46" (OuterVolumeSpecName: "kube-api-access-l5m46") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "kube-api-access-l5m46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:06.694848 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.694833 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:06.694893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.694854 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" (UID: "065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:14:06.794167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.794145 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.794167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.794165 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.794302 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.794175 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5m46\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-kube-api-access-l5m46\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:06.794302 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:06.794184 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:07.579215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:07.579184 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h" Sep 10 21:14:07.601775 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:07.601745 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:14:07.609468 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:07.609444 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-21de0f-openshift-default-5d987c799b-crn5h"] Sep 10 21:14:08.817331 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:08.817298 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" path="/var/lib/kubelet/pods/065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d/volumes" Sep 10 21:14:21.607481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607448 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:14:21.607915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607877 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:21.607915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607894 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:21.607915 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607915 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" containerName="istio-proxy" Sep 10 21:14:21.608020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607920 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" containerName="istio-proxy" Sep 10 21:14:21.608020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607978 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="065ee6ba-6af3-4fe3-a677-f0f8fbfeab9d" containerName="istio-proxy" Sep 10 21:14:21.608020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.607988 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:21.612429 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.612412 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.616254 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.616235 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6e2e41d-openshift-default-dockercfg-z7h4h\"" Sep 10 21:14:21.636218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.636191 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:14:21.731024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.730989 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtsw\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731029 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731486 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.731486 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.731414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.832663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.832832 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtsw\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.832832 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.832832 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833015 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833015 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833015 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.832975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833015 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833006 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833220 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833220 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833220 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833390 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833433 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833404 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.833649 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.833603 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.835087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.835068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.835250 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.835234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.841766 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.841744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.842048 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.842031 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtsw\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw\") pod \"e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:21.922233 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:21.922180 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:22.200122 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.200022 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:14:22.207485 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:14:22.207447 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c52c4eb_bab5_4d35_810f_5d246e4bd79f.slice/crio-7471ce4ea52d9a12114ba9158f9e1f57f5702ba174768891333471b6c7c5409a WatchSource:0}: Error finding container 7471ce4ea52d9a12114ba9158f9e1f57f5702ba174768891333471b6c7c5409a: Status 404 returned error can't find the container with id 7471ce4ea52d9a12114ba9158f9e1f57f5702ba174768891333471b6c7c5409a Sep 10 21:14:22.210076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.210035 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:22.210193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.210151 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:22.210255 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.210199 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:22.667178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.667145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerStarted","Data":"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641"} Sep 10 21:14:22.667571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.667204 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerStarted","Data":"7471ce4ea52d9a12114ba9158f9e1f57f5702ba174768891333471b6c7c5409a"} Sep 10 21:14:22.696517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.696458 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podStartSLOduration=1.6964388860000001 podStartE2EDuration="1.696438886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:22.692817315 +0000 UTC m=+946.401111702" watchObservedRunningTime="2026-09-10 21:14:22.696438886 +0000 UTC m=+946.404733273" Sep 10 21:14:22.922938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:22.922857 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:23.930994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:23.930959 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:24.678121 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:24.678089 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:24.678938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:24.678918 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:43.781223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.781188 2585 generic.go:356] "Generic (PLEG): container finished" podID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerID="7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641" exitCode=0 Sep 10 21:14:43.781694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.781264 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerDied","Data":"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641"} Sep 10 21:14:43.781755 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.781741 2585 scope.go:117] "RemoveContainer" containerID="7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641" Sep 10 21:14:43.782783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.782757 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:43.782844 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.782823 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:43.782889 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.782854 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:14:43.919710 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.919669 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:14:43.919905 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:43.919883 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" containerID="cri-o://8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d" gracePeriod=30 Sep 10 21:14:44.265551 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.265459 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw"] Sep 10 21:14:44.265876 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.265862 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:44.265927 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.265878 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:44.265962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.265958 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b253e4c-79f2-42a5-a239-41ef4514375b" containerName="istio-proxy" Sep 10 21:14:44.269069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.269052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:44.271796 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.271767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:14:44.271921 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.271905 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:14:44.281561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.281527 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw"] Sep 10 21:14:44.378089 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.378049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw\" (UID: \"7a513974-8205-4318-936f-2716916aa752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:44.479091 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.479052 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw\" (UID: \"7a513974-8205-4318-936f-2716916aa752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:44.480263 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.480066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw\" (UID: \"7a513974-8205-4318-936f-2716916aa752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:44.585564 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.585529 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:44.805320 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.805284 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerStarted","Data":"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591"} Sep 10 21:14:44.883308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.883206 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw"] Sep 10 21:14:44.923211 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.923186 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:44.923481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.923451 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 21:14:44.923635 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:44.923510 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 21:14:45.813309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:45.813277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" event={"ID":"7a513974-8205-4318-936f-2716916aa752","Type":"ContainerStarted","Data":"ab4ce81a2d74017bb2e130e3240a543fef3c4fbe5c44304d722ae90bb77f23a4"} Sep 10 21:14:45.813838 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:45.813328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" event={"ID":"7a513974-8205-4318-936f-2716916aa752","Type":"ContainerStarted","Data":"89a850cd205bb14bbaec52046a36e6908f39c167ae8125b4c1425184fd8158d6"} Sep 10 21:14:45.863850 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:45.863793 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" podStartSLOduration=1.8637765929999999 podStartE2EDuration="1.863776593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:45.862696755 +0000 UTC m=+969.570991141" watchObservedRunningTime="2026-09-10 21:14:45.863776593 +0000 UTC m=+969.572070992" Sep 10 21:14:45.923451 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:45.923426 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 21:14:45.923558 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:45.923486 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 21:14:46.819446 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:46.819414 2585 generic.go:356] "Generic (PLEG): container finished" podID="7a513974-8205-4318-936f-2716916aa752" containerID="ab4ce81a2d74017bb2e130e3240a543fef3c4fbe5c44304d722ae90bb77f23a4" exitCode=0 Sep 10 21:14:46.819848 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:46.819490 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" event={"ID":"7a513974-8205-4318-936f-2716916aa752","Type":"ContainerDied","Data":"ab4ce81a2d74017bb2e130e3240a543fef3c4fbe5c44304d722ae90bb77f23a4"} Sep 10 21:14:46.923235 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:46.923198 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 21:14:46.923420 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:46.923254 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 21:14:47.923231 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:47.923196 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 21:14:47.923602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:47.923267 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 21:14:47.997929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:47.997898 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:48.025034 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.024998 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca\") pod \"7a513974-8205-4318-936f-2716916aa752\" (UID: \"7a513974-8205-4318-936f-2716916aa752\") " Sep 10 21:14:48.025279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.025260 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a513974-8205-4318-936f-2716916aa752" (UID: "7a513974-8205-4318-936f-2716916aa752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:14:48.025349 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.025334 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a513974-8205-4318-936f-2716916aa752-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:48.834058 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.834017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" event={"ID":"7a513974-8205-4318-936f-2716916aa752","Type":"ContainerDied","Data":"89a850cd205bb14bbaec52046a36e6908f39c167ae8125b4c1425184fd8158d6"} Sep 10 21:14:48.834226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.834071 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89a850cd205bb14bbaec52046a36e6908f39c167ae8125b4c1425184fd8158d6" Sep 10 21:14:48.834226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.834042 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw" Sep 10 21:14:48.923216 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.923175 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 21:14:48.923358 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:48.923248 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 21:14:49.207315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.207288 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:14:49.339800 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339770 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.339800 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339806 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339849 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339918 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339958 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rfj5\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.339997 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340029 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:49.340364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340157 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340196 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket\") pod \"957fdd18-3e09-4eb7-b737-380fb84874ce\" (UID: \"957fdd18-3e09-4eb7-b737-380fb84874ce\") " Sep 10 21:14:49.340364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340232 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:49.340364 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340269 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:14:49.340582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340378 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:49.340582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340521 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:49.340731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340606 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.340731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340643 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957fdd18-3e09-4eb7-b737-380fb84874ce-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.340731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340656 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.340731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340668 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.340731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.340682 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.342154 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.342129 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:14:49.342264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.342146 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5" (OuterVolumeSpecName: "kube-api-access-2rfj5") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "kube-api-access-2rfj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:49.342264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.342218 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:14:49.342336 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.342320 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "957fdd18-3e09-4eb7-b737-380fb84874ce" (UID: "957fdd18-3e09-4eb7-b737-380fb84874ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:49.441321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.441285 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.441321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.441308 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfj5\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-kube-api-access-2rfj5\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.441321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.441318 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.441321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.441326 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957fdd18-3e09-4eb7-b737-380fb84874ce-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:14:49.841446 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.841410 2585 generic.go:356] "Generic (PLEG): container finished" podID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerID="8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d" exitCode=0 Sep 10 21:14:49.841655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.841465 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" Sep 10 21:14:49.841655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.841495 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerDied","Data":"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d"} Sep 10 21:14:49.841655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.841560 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4" event={"ID":"957fdd18-3e09-4eb7-b737-380fb84874ce","Type":"ContainerDied","Data":"bdda57985e736587616b79c36ef8914defb54c2b2db6b93af1c1c576d4fa860b"} Sep 10 21:14:49.841655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.841600 2585 scope.go:117] "RemoveContainer" containerID="8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d" Sep 10 21:14:49.852254 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.852236 2585 scope.go:117] "RemoveContainer" containerID="8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c" Sep 10 21:14:49.861184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.861163 2585 scope.go:117] "RemoveContainer" containerID="8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d" Sep 10 21:14:49.861464 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:14:49.861445 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d\": container with ID starting with 8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d not found: ID does not exist" containerID="8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d" Sep 10 21:14:49.861548 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.861472 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d"} err="failed to get container status \"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d\": rpc error: code = NotFound desc = could not find container \"8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d\": container with ID starting with 8f3b2a932a5178ce55341dbdb8896e132a9a41324e8d37848c23620a8602ef9d not found: ID does not exist" Sep 10 21:14:49.861548 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.861490 2585 scope.go:117] "RemoveContainer" containerID="8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c" Sep 10 21:14:49.861773 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:14:49.861755 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c\": container with ID starting with 8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c not found: ID does not exist" containerID="8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c" Sep 10 21:14:49.861816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.861779 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c"} err="failed to get container status \"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c\": rpc error: code = NotFound desc = could not find container \"8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c\": container with ID starting with 8949af3db9f85352775daa15041219daa5ce6ceffc7e4607ae235adee69a010c not found: ID does not exist" Sep 10 21:14:49.865000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.864979 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:14:49.869444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.869421 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-76b274-openshift-default-5d877c9fdb-vnml4"] Sep 10 21:14:49.984008 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.983978 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:49.984375 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.984257 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:49.984950 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:49.984930 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:14:50.818719 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:50.818685 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" path="/var/lib/kubelet/pods/957fdd18-3e09-4eb7-b737-380fb84874ce/volumes" Sep 10 21:14:55.441974 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:55.441939 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw"] Sep 10 21:14:55.443783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:55.443760 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31e7cb-5lvcw"] Sep 10 21:14:56.817924 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:56.817891 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a513974-8205-4318-936f-2716916aa752" path="/var/lib/kubelet/pods/7a513974-8205-4318-936f-2716916aa752/volumes" Sep 10 21:14:57.468245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468215 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x"] Sep 10 21:14:57.468632 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468619 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a513974-8205-4318-936f-2716916aa752" containerName="revoke-keys" Sep 10 21:14:57.468685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468635 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a513974-8205-4318-936f-2716916aa752" containerName="revoke-keys" Sep 10 21:14:57.468685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468647 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.468685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468653 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.468685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468659 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.468685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468665 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.468838 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468727 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.468838 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468735 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a513974-8205-4318-936f-2716916aa752" containerName="revoke-keys" Sep 10 21:14:57.468838 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.468741 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="957fdd18-3e09-4eb7-b737-380fb84874ce" containerName="istio-proxy" Sep 10 21:14:57.473034 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.473018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:14:57.476351 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.476329 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:14:57.476467 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.476363 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:14:57.489211 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.489188 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x"] Sep 10 21:14:57.617921 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.617889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x\" (UID: \"67f0cdc6-632f-4a49-96f0-c5155b3f07a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:14:57.718739 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.718653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x\" (UID: \"67f0cdc6-632f-4a49-96f0-c5155b3f07a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:14:57.719251 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.719232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x\" (UID: \"67f0cdc6-632f-4a49-96f0-c5155b3f07a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:14:57.782661 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:57.782637 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:14:58.089709 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:58.089679 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x"] Sep 10 21:14:58.093402 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:14:58.093369 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f0cdc6_632f_4a49_96f0_c5155b3f07a2.slice/crio-ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd WatchSource:0}: Error finding container ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd: Status 404 returned error can't find the container with id ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd Sep 10 21:14:58.890397 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:58.890367 2585 generic.go:356] "Generic (PLEG): container finished" podID="67f0cdc6-632f-4a49-96f0-c5155b3f07a2" containerID="419c531c56f6bf9796a2e2453efe08ddca745d566e35e67503ce8dc919bd8f87" exitCode=0 Sep 10 21:14:58.890567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:58.890454 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" event={"ID":"67f0cdc6-632f-4a49-96f0-c5155b3f07a2","Type":"ContainerDied","Data":"419c531c56f6bf9796a2e2453efe08ddca745d566e35e67503ce8dc919bd8f87"} Sep 10 21:14:58.890567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:14:58.890516 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" event={"ID":"67f0cdc6-632f-4a49-96f0-c5155b3f07a2","Type":"ContainerStarted","Data":"ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd"} Sep 10 21:15:00.031746 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.031720 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:15:00.136145 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.136117 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca\") pod \"67f0cdc6-632f-4a49-96f0-c5155b3f07a2\" (UID: \"67f0cdc6-632f-4a49-96f0-c5155b3f07a2\") " Sep 10 21:15:00.136489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.136461 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67f0cdc6-632f-4a49-96f0-c5155b3f07a2" (UID: "67f0cdc6-632f-4a49-96f0-c5155b3f07a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:15:00.148652 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.148573 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96"] Sep 10 21:15:00.149239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.149221 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f0cdc6-632f-4a49-96f0-c5155b3f07a2" containerName="revoke-keys" Sep 10 21:15:00.149281 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.149244 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f0cdc6-632f-4a49-96f0-c5155b3f07a2" containerName="revoke-keys" Sep 10 21:15:00.149353 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.149343 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f0cdc6-632f-4a49-96f0-c5155b3f07a2" containerName="revoke-keys" Sep 10 21:15:00.153235 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.153208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" Sep 10 21:15:00.159499 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.159475 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96"] Sep 10 21:15:00.165886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.165866 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" Sep 10 21:15:00.237303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.237275 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f0cdc6-632f-4a49-96f0-c5155b3f07a2-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:00.457445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.457416 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96"] Sep 10 21:15:00.459289 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:15:00.459257 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc197ab4_5383_49e3_b660_a44f7fde9689.slice/crio-6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe WatchSource:0}: Error finding container 6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe: Status 404 returned error can't find the container with id 6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe Sep 10 21:15:00.902261 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.902223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" event={"ID":"bc197ab4-5383-49e3-b660-a44f7fde9689","Type":"ContainerStarted","Data":"02121e8e0711955900ffb60349ee8fbb95755555fc28670ba5ec53df32b1ce8c"} Sep 10 21:15:00.902437 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.902285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" event={"ID":"bc197ab4-5383-49e3-b660-a44f7fde9689","Type":"ContainerStarted","Data":"6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe"} Sep 10 21:15:00.903580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.903556 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" event={"ID":"67f0cdc6-632f-4a49-96f0-c5155b3f07a2","Type":"ContainerDied","Data":"ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd"} Sep 10 21:15:00.903580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.903579 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x" Sep 10 21:15:00.903739 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.903593 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7ce4e28350e420d9ea3aa6f7a07c1ec95072291022ce2c5a4be70b45e555cd" Sep 10 21:15:00.921736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:00.921692 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" podStartSLOduration=0.921679412 podStartE2EDuration="921.679412ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:00.918489644 +0000 UTC m=+984.626784030" watchObservedRunningTime="2026-09-10 21:15:00.921679412 +0000 UTC m=+984.629973801" Sep 10 21:15:01.910283 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:01.910250 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc197ab4-5383-49e3-b660-a44f7fde9689" containerID="02121e8e0711955900ffb60349ee8fbb95755555fc28670ba5ec53df32b1ce8c" exitCode=0 Sep 10 21:15:01.910696 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:01.910313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" event={"ID":"bc197ab4-5383-49e3-b660-a44f7fde9689","Type":"ContainerDied","Data":"02121e8e0711955900ffb60349ee8fbb95755555fc28670ba5ec53df32b1ce8c"} Sep 10 21:15:02.632627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:02.632587 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:15:02.632833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:02.632805 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" containerID="cri-o://5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9" gracePeriod=30 Sep 10 21:15:03.036223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:03.036202 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" Sep 10 21:15:03.922650 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:03.922599 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" Sep 10 21:15:03.922833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:03.922598 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-trb96" event={"ID":"bc197ab4-5383-49e3-b660-a44f7fde9689","Type":"ContainerDied","Data":"6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe"} Sep 10 21:15:03.922833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:03.922751 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6415d0832303d7d2748d9f69d59b720dec59149acaeb8f655fefe782ccff77fe" Sep 10 21:15:07.882102 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.882079 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:15:07.944953 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.944875 2585 generic.go:356] "Generic (PLEG): container finished" podID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerID="5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9" exitCode=0 Sep 10 21:15:07.944953 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.944929 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" Sep 10 21:15:07.945161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.944956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerDied","Data":"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9"} Sep 10 21:15:07.945161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.945005 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx" event={"ID":"c3367312-ecd3-41d3-bd98-b85b459771b8","Type":"ContainerDied","Data":"8bd537ee556da3601f6999f4bf243d49f124ec24af014ae791ee825728677905"} Sep 10 21:15:07.945161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.945030 2585 scope.go:117] "RemoveContainer" containerID="5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9" Sep 10 21:15:07.953812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.953796 2585 scope.go:117] "RemoveContainer" containerID="e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263" Sep 10 21:15:07.961629 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.961595 2585 scope.go:117] "RemoveContainer" containerID="5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9" Sep 10 21:15:07.961882 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:15:07.961860 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9\": container with ID starting with 5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9 not found: ID does not exist" containerID="5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9" Sep 10 21:15:07.961969 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.961889 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9"} err="failed to get container status \"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9\": rpc error: code = NotFound desc = could not find container \"5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9\": container with ID starting with 5e2270ec53608a5b6b173d64b9b5bb9bc01d1aaac7bfcb6b453cc27cdf9e50f9 not found: ID does not exist" Sep 10 21:15:07.961969 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.961905 2585 scope.go:117] "RemoveContainer" containerID="e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263" Sep 10 21:15:07.962146 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:15:07.962131 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263\": container with ID starting with e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263 not found: ID does not exist" containerID="e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263" Sep 10 21:15:07.962191 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:07.962149 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263"} err="failed to get container status \"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263\": rpc error: code = NotFound desc = could not find container \"e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263\": container with ID starting with e7ce4dbcb887900f9df49aea2e014111c3f41c1377dedc35dd84a69abcf7b263 not found: ID does not exist" Sep 10 21:15:08.009671 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009646 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.009826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009700 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj4n5\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.009826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009722 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.009826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009768 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.009826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009798 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.009826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009814 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.010096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009861 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.010096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009889 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.010096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.009935 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token\") pod \"c3367312-ecd3-41d3-bd98-b85b459771b8\" (UID: \"c3367312-ecd3-41d3-bd98-b85b459771b8\") " Sep 10 21:15:08.010260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010128 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:08.010260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010045 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:15:08.010260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010155 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:08.010422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010269 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.010422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010329 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.010422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010339 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:08.010422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010346 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3367312-ecd3-41d3-bd98-b85b459771b8-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.010422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.010402 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:08.011931 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.011907 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5" (OuterVolumeSpecName: "kube-api-access-kj4n5") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "kube-api-access-kj4n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:15:08.012106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.012092 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:15:08.012380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.012357 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:15:08.012380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.012362 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3367312-ecd3-41d3-bd98-b85b459771b8" (UID: "c3367312-ecd3-41d3-bd98-b85b459771b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:08.112067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112033 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.112067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112068 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.112273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112081 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.112273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112091 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.112273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112101 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4n5\" (UniqueName: \"kubernetes.io/projected/c3367312-ecd3-41d3-bd98-b85b459771b8-kube-api-access-kj4n5\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.112273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.112110 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3367312-ecd3-41d3-bd98-b85b459771b8-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:08.268378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.268340 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:15:08.275421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.275396 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-31e7cb-openshift-default-79bb4c5598-zhnfx"] Sep 10 21:15:08.377025 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.376990 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x"] Sep 10 21:15:08.378784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.378760 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6e2e41d-dmh6x"] Sep 10 21:15:08.818093 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.818064 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f0cdc6-632f-4a49-96f0-c5155b3f07a2" path="/var/lib/kubelet/pods/67f0cdc6-632f-4a49-96f0-c5155b3f07a2/volumes" Sep 10 21:15:08.818478 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:08.818464 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" path="/var/lib/kubelet/pods/c3367312-ecd3-41d3-bd98-b85b459771b8/volumes" Sep 10 21:15:15.547537 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:15.547506 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:15:15.547930 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:15.547719 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" containerID="cri-o://7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591" gracePeriod=30 Sep 10 21:15:19.986895 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:19.986858 2585 patch_prober.go:28] interesting pod/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 21:15:19.987256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:19.986924 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:15:20.798051 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.798025 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:15:20.822342 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822279 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822342 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822316 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mtsw\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822342 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822333 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822351 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822389 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822428 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822454 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822554 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822852 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822579 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket\") pod \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\" (UID: \"0c52c4eb-bab5-4d35-810f-5d246e4bd79f\") " Sep 10 21:15:20.822852 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822585 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:20.822852 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822840 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:20.823027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.822774 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:20.823082 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.823040 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:20.823127 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.823107 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:15:20.823309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.823266 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.823309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.823288 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.824831 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.824799 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:15:20.825222 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.825201 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:15:20.825377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.825331 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw" (OuterVolumeSpecName: "kube-api-access-8mtsw") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "kube-api-access-8mtsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:15:20.825377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.825362 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c52c4eb-bab5-4d35-810f-5d246e4bd79f" (UID: "0c52c4eb-bab5-4d35-810f-5d246e4bd79f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:15:20.923966 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.923938 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.923966 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.923964 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.924119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.923980 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtsw\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-kube-api-access-8mtsw\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.924119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.923993 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.924119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.924004 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.924119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.924015 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:20.924119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:20.924026 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c52c4eb-bab5-4d35-810f-5d246e4bd79f-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:15:21.019442 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.019414 2585 generic.go:356] "Generic (PLEG): container finished" podID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerID="7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591" exitCode=0 Sep 10 21:15:21.019814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.019460 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" Sep 10 21:15:21.019814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.019500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerDied","Data":"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591"} Sep 10 21:15:21.019814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.019548 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv" event={"ID":"0c52c4eb-bab5-4d35-810f-5d246e4bd79f","Type":"ContainerDied","Data":"7471ce4ea52d9a12114ba9158f9e1f57f5702ba174768891333471b6c7c5409a"} Sep 10 21:15:21.019814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.019568 2585 scope.go:117] "RemoveContainer" containerID="7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591" Sep 10 21:15:21.028844 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.028827 2585 scope.go:117] "RemoveContainer" containerID="7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641" Sep 10 21:15:21.036575 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.036561 2585 scope.go:117] "RemoveContainer" containerID="7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591" Sep 10 21:15:21.036866 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:15:21.036845 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591\": container with ID starting with 7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591 not found: ID does not exist" containerID="7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591" Sep 10 21:15:21.036935 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.036872 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591"} err="failed to get container status \"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591\": rpc error: code = NotFound desc = could not find container \"7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591\": container with ID starting with 7a8960a9c582b6e36908215e4bc28378343e253eeee46f06647e01ab2e195591 not found: ID does not exist" Sep 10 21:15:21.036935 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.036889 2585 scope.go:117] "RemoveContainer" containerID="7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641" Sep 10 21:15:21.037139 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:15:21.037123 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641\": container with ID starting with 7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641 not found: ID does not exist" containerID="7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641" Sep 10 21:15:21.037176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.037146 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641"} err="failed to get container status \"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641\": rpc error: code = NotFound desc = could not find container \"7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641\": container with ID starting with 7db8fc38fee182faeebf34b7fa504df36ea1ec987bfe37327a680a0ca4204641 not found: ID does not exist" Sep 10 21:15:21.043421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.043398 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:15:21.049853 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:21.049833 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6e2e41d-openshift-default-74bf689784-sbwtv"] Sep 10 21:15:22.823085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:22.823056 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" path="/var/lib/kubelet/pods/0c52c4eb-bab5-4d35-810f-5d246e4bd79f/volumes" Sep 10 21:15:29.911968 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.911933 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:15:29.912346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912333 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:29.912346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912347 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912367 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912372 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912378 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc197ab4-5383-49e3-b660-a44f7fde9689" containerName="cleanup" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912384 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc197ab4-5383-49e3-b660-a44f7fde9689" containerName="cleanup" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912394 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912425 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912399 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912454 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912461 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:29.912624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912470 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c52c4eb-bab5-4d35-810f-5d246e4bd79f" containerName="istio-proxy" Sep 10 21:15:29.912624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.912478 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc197ab4-5383-49e3-b660-a44f7fde9689" containerName="cleanup" Sep 10 21:15:29.916924 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.916908 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:29.919993 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.919972 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-290e03-openshift-default-dockercfg-8fncc\"" Sep 10 21:15:29.942431 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:29.942409 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:15:30.004205 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004162 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4pz\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004373 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004373 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004373 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004427 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004449 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004597 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.004655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.004639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105192 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105200 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105257 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105334 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105566 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105352 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4pz\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105566 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105715 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105712 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.105771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105729 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.106001 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.105980 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.106100 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.106076 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.106100 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.106090 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.107541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.107521 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.107712 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.107695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.113539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.113516 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.113806 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.113789 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4pz\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz\") pod \"e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.226906 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.226845 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:30.507213 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.507177 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:15:30.508047 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:15:30.508020 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0955c7ed_ef80_44dc_b859_d48bcdc601d8.slice/crio-1ac2a08aa27f767bf15c5ba3d87e60566de5a8d8fd515b0578738de247dbd78f WatchSource:0}: Error finding container 1ac2a08aa27f767bf15c5ba3d87e60566de5a8d8fd515b0578738de247dbd78f: Status 404 returned error can't find the container with id 1ac2a08aa27f767bf15c5ba3d87e60566de5a8d8fd515b0578738de247dbd78f Sep 10 21:15:30.510239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.510209 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:30.510298 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.510268 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:30.510332 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:30.510299 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:31.073126 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:31.073095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerStarted","Data":"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649"} Sep 10 21:15:31.073491 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:31.073145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerStarted","Data":"1ac2a08aa27f767bf15c5ba3d87e60566de5a8d8fd515b0578738de247dbd78f"} Sep 10 21:15:31.097595 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:31.097538 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podStartSLOduration=2.097522294 podStartE2EDuration="2.097522294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:31.095238206 +0000 UTC m=+1014.803532593" watchObservedRunningTime="2026-09-10 21:15:31.097522294 +0000 UTC m=+1014.805816680" Sep 10 21:15:31.227069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:31.227031 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:31.231541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:31.231519 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:32.079227 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:32.079201 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:32.080167 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:32.080148 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:35.835358 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.835327 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:15:35.835823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.835807 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:35.835869 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.835825 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:35.835924 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.835915 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3367312-ecd3-41d3-bd98-b85b459771b8" containerName="istio-proxy" Sep 10 21:15:35.839041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.839019 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.842427 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.842407 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-755926-openshift-default-dockercfg-jldg6\"" Sep 10 21:15:35.853659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.852735 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:15:35.959596 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959560 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959596 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnsj\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959801 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959930 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.959930 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959890 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.960038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:35.960038 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:35.959955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061107 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061107 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnsj\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061292 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061512 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061550 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061575 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.061983 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.061829 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.062046 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.062017 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.063752 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.063733 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.063923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.063903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.071118 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.071093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnsj\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.071756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.071737 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token\") pod \"e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.149767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.149715 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:36.424511 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.424437 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:15:36.426862 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:15:36.426831 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode018a895_59a0_4c6d_b9f7_0df36808013d.slice/crio-d92dd5abba9ab206609dd1863f9cfc46c230f0fd790e37d9223f89b063a2a129 WatchSource:0}: Error finding container d92dd5abba9ab206609dd1863f9cfc46c230f0fd790e37d9223f89b063a2a129: Status 404 returned error can't find the container with id d92dd5abba9ab206609dd1863f9cfc46c230f0fd790e37d9223f89b063a2a129 Sep 10 21:15:36.429080 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.429050 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:36.429193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.429135 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:36.429193 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:36.429168 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:37.107837 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:37.107797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerStarted","Data":"5483838bf85fb2dab16149ce8280208f9940caa5d7303e07af698ad527f28726"} Sep 10 21:15:37.108230 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:37.107855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerStarted","Data":"d92dd5abba9ab206609dd1863f9cfc46c230f0fd790e37d9223f89b063a2a129"} Sep 10 21:15:37.132524 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:37.132431 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podStartSLOduration=2.132419211 podStartE2EDuration="2.132419211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:37.131690596 +0000 UTC m=+1020.839984982" watchObservedRunningTime="2026-09-10 21:15:37.132419211 +0000 UTC m=+1020.840713653" Sep 10 21:15:37.150812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:37.150788 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:37.155637 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:37.155596 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:38.113159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:38.113125 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:38.113991 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:38.113968 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:48.132812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:48.132769 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 21:15:48.133210 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:48.132843 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 21:15:51.186320 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.186283 2585 generic.go:356] "Generic (PLEG): container finished" podID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerID="e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649" exitCode=0 Sep 10 21:15:51.187979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.186945 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerDied","Data":"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649"} Sep 10 21:15:51.187979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.187589 2585 scope.go:117] "RemoveContainer" containerID="e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649" Sep 10 21:15:51.188985 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.188941 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:51.189106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.189042 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:51.189106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:51.189090 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:52.199137 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:52.199092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerStarted","Data":"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589"} Sep 10 21:15:52.227985 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:52.227942 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:52.228800 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:52.228774 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:52.228899 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:52.228824 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:53.228269 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:53.228230 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:53.228676 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:53.228302 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:54.154815 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:54.154771 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 21:15:54.155022 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:54.154842 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 21:15:54.228229 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:54.228191 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:54.228395 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:54.228257 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:55.227914 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:55.227876 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:55.228117 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:55.227939 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:56.228139 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:56.228101 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:56.228518 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:56.228166 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:57.227277 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.227247 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 21:15:57.227532 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.227298 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 21:15:57.228540 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.228516 2585 generic.go:356] "Generic (PLEG): container finished" podID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerID="5483838bf85fb2dab16149ce8280208f9940caa5d7303e07af698ad527f28726" exitCode=0 Sep 10 21:15:57.228912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.228637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerDied","Data":"5483838bf85fb2dab16149ce8280208f9940caa5d7303e07af698ad527f28726"} Sep 10 21:15:57.229053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.229038 2585 scope.go:117] "RemoveContainer" containerID="5483838bf85fb2dab16149ce8280208f9940caa5d7303e07af698ad527f28726" Sep 10 21:15:57.230084 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.230060 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:57.230137 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.230124 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:57.230172 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:57.230157 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:58.231270 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.231242 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:58.231683 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.231553 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:58.232429 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.232408 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:15:58.237318 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.237295 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerStarted","Data":"831d5ce91458b24844beb2afb9fbc6125af2c8ff50a236ba07a8429b10f2c19c"} Sep 10 21:15:58.528380 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.528347 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:15:58.532009 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.531984 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.535440 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.535412 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9f71ac-openshift-default-dockercfg-b7rvq\"" Sep 10 21:15:58.554297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.554270 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:15:58.601159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601115 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601155 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601181 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsm7\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601275 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601325 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601773 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601473 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.601773 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.601567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702416 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702470 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702494 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsm7\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.702756 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.703159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702881 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.703159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.702946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.703159 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.703125 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.703383 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.703364 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.703561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.703538 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.705089 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.705066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.705148 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.705096 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.711863 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.711840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsm7\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.711971 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.711952 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token\") pod \"e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:58.843266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:58.843181 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:15:59.143503 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.143477 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:15:59.144818 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:15:59.144788 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caef289_5318_45d0_952a_26e4e82d47fc.slice/crio-dd25f631ac6ae81ad33eca54e5e8359c3ce6fce7472f9b865709eb9fc7bd0ed5 WatchSource:0}: Error finding container dd25f631ac6ae81ad33eca54e5e8359c3ce6fce7472f9b865709eb9fc7bd0ed5: Status 404 returned error can't find the container with id dd25f631ac6ae81ad33eca54e5e8359c3ce6fce7472f9b865709eb9fc7bd0ed5 Sep 10 21:15:59.147161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.147124 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:59.147264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.147200 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:59.147264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.147229 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:15:59.150264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.150244 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:15:59.150469 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.150445 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 21:15:59.150559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.150492 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 21:15:59.245064 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.245029 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerStarted","Data":"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5"} Sep 10 21:15:59.245417 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.245094 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerStarted","Data":"dd25f631ac6ae81ad33eca54e5e8359c3ce6fce7472f9b865709eb9fc7bd0ed5"} Sep 10 21:15:59.267297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.267243 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podStartSLOduration=1.267222229 podStartE2EDuration="1.267222229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:59.265337799 +0000 UTC m=+1042.973632182" watchObservedRunningTime="2026-09-10 21:15:59.267222229 +0000 UTC m=+1042.975516617" Sep 10 21:15:59.844051 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:15:59.844011 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:00.151115 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:00.151030 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 21:16:00.151115 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:00.151082 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 21:16:00.848991 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:00.848965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:01.150321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:01.150217 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 21:16:01.150321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:01.150291 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 21:16:01.255347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:01.255320 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:01.256263 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:01.256244 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:02.150271 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:02.150228 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 21:16:02.150724 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:02.150290 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 21:16:03.150094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:03.150050 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 21:16:03.150282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:03.150101 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 21:16:04.154309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:04.154234 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:16:04.154753 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:04.154655 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:16:04.155477 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:04.155456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:16:20.356058 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.356023 2585 generic.go:356] "Generic (PLEG): container finished" podID="4caef289-5318-45d0-952a-26e4e82d47fc" containerID="0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5" exitCode=0 Sep 10 21:16:20.356454 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.356101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerDied","Data":"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5"} Sep 10 21:16:20.356526 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.356510 2585 scope.go:117] "RemoveContainer" containerID="0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5" Sep 10 21:16:20.357568 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.357529 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:20.357691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.357605 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:20.357691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:20.357652 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:21.366495 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:21.366449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerStarted","Data":"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b"} Sep 10 21:16:21.844150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:21.844112 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:21.844347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:21.844325 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 21:16:21.844409 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:21.844374 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 21:16:22.844178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:22.844144 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 21:16:22.844553 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:22.844196 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 21:16:23.843882 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:23.843851 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 21:16:23.844066 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:23.843906 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 21:16:24.843953 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:24.843916 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 21:16:24.844427 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:24.843976 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 21:16:25.843839 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:25.843800 2585 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 21:16:25.844011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:25.843864 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 21:16:26.848235 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:26.848200 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:26.848597 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:26.848484 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:26.849186 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:26.849171 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:16:35.912385 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.912307 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:16:35.915714 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.915698 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.920000 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.919976 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8b5fd067-openshift-default-dockercfg-bgnmx\"" Sep 10 21:16:35.932488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.932463 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:16:35.975124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975098 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975167 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975510 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975412 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:35.975510 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:35.975432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fhn\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076166 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076128 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076261 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fhn\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076578 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.076845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.077122 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.076841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.077122 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.077014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.077274 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.077256 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.078558 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.078532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.078833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.078815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.087308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.087281 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.087533 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.087517 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fhn\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn\") pod \"e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.225494 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.225435 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:36.506595 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.506566 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:16:36.508325 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:16:36.508301 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efad980_8873_4cb6_a600_f3de36007995.slice/crio-b4e1cfa29c71b72d88ccf8209cadbff661dd1c91f1185102fc80aed0ed7c3bf1 WatchSource:0}: Error finding container b4e1cfa29c71b72d88ccf8209cadbff661dd1c91f1185102fc80aed0ed7c3bf1: Status 404 returned error can't find the container with id b4e1cfa29c71b72d88ccf8209cadbff661dd1c91f1185102fc80aed0ed7c3bf1 Sep 10 21:16:36.510642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.510591 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:36.510724 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.510674 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:36.510724 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:36.510700 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:16:37.472827 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:37.472791 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerStarted","Data":"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805"} Sep 10 21:16:37.473202 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:37.472846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerStarted","Data":"b4e1cfa29c71b72d88ccf8209cadbff661dd1c91f1185102fc80aed0ed7c3bf1"} Sep 10 21:16:37.498058 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:37.498012 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podStartSLOduration=2.497999076 podStartE2EDuration="2.497999076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:37.495485395 +0000 UTC m=+1081.203779781" watchObservedRunningTime="2026-09-10 21:16:37.497999076 +0000 UTC m=+1081.206293527" Sep 10 21:16:38.225763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:38.225723 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:38.230736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:38.230713 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:38.479004 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:38.478916 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:16:38.480325 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:16:38.480305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:17:01.613136 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.613058 2585 generic.go:356] "Generic (PLEG): container finished" podID="7efad980-8873-4cb6-a600-f3de36007995" containerID="6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805" exitCode=0 Sep 10 21:17:01.613136 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.613123 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerDied","Data":"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805"} Sep 10 21:17:01.613545 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.613505 2585 scope.go:117] "RemoveContainer" containerID="6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805" Sep 10 21:17:01.614708 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.614679 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:17:01.614816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.614791 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:17:01.614886 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:01.614844 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:17:02.620359 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:02.620325 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerStarted","Data":"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53"} Sep 10 21:17:03.225814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:03.225773 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:17:03.226106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:03.226082 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:17:03.226187 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:03.226167 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:17:04.226820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:04.226783 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:17:04.227236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:04.226868 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:17:05.226293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:05.226253 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:17:05.226478 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:05.226321 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:17:06.226434 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:06.226403 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:17:06.226824 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:06.226464 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:17:07.226035 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:07.225995 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:17:07.226235 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:07.226064 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:17:08.229740 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:08.229711 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:17:08.230120 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:08.229978 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:17:08.230544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:08.230527 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:17:33.219573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.219539 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd"] Sep 10 21:17:33.223319 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.223301 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:33.225864 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.225844 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:17:33.225984 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.225897 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:17:33.231555 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.231529 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd"] Sep 10 21:17:33.332488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.332456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd\" (UID: \"34446a73-78f1-4f18-a0f3-06befece1803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:33.433692 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.433649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd\" (UID: \"34446a73-78f1-4f18-a0f3-06befece1803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:33.434245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.434222 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd\" (UID: \"34446a73-78f1-4f18-a0f3-06befece1803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:33.534647 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.534583 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:33.812858 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:33.812835 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd"] Sep 10 21:17:33.815047 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:17:33.815021 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34446a73_78f1_4f18_a0f3_06befece1803.slice/crio-8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd WatchSource:0}: Error finding container 8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd: Status 404 returned error can't find the container with id 8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd Sep 10 21:17:34.795700 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:34.795668 2585 generic.go:356] "Generic (PLEG): container finished" podID="34446a73-78f1-4f18-a0f3-06befece1803" containerID="b2428b41f6d4eef225f4412c634c91a6dad17b5ef87771b502ee9c496a0f2c62" exitCode=0 Sep 10 21:17:34.796056 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:34.795755 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" event={"ID":"34446a73-78f1-4f18-a0f3-06befece1803","Type":"ContainerDied","Data":"b2428b41f6d4eef225f4412c634c91a6dad17b5ef87771b502ee9c496a0f2c62"} Sep 10 21:17:34.796056 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:34.795801 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" event={"ID":"34446a73-78f1-4f18-a0f3-06befece1803","Type":"ContainerStarted","Data":"8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd"} Sep 10 21:17:35.936787 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:35.936763 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:36.058867 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.058788 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca\") pod \"34446a73-78f1-4f18-a0f3-06befece1803\" (UID: \"34446a73-78f1-4f18-a0f3-06befece1803\") " Sep 10 21:17:36.059199 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.059173 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34446a73-78f1-4f18-a0f3-06befece1803" (UID: "34446a73-78f1-4f18-a0f3-06befece1803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:17:36.160122 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.160094 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34446a73-78f1-4f18-a0f3-06befece1803-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:17:36.808149 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.808122 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" Sep 10 21:17:36.808312 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.808118 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd" event={"ID":"34446a73-78f1-4f18-a0f3-06befece1803","Type":"ContainerDied","Data":"8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd"} Sep 10 21:17:36.808312 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.808279 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ceba44da9ebfec5974e5c084c10ab22309e8b953ce0dc17b3633812dc8fb0fd" Sep 10 21:17:36.892731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.892701 2585 scope.go:117] "RemoveContainer" containerID="96329bb145bfb172f54c2507efc339a246e9b1b0b8760a44802aa99c454603b0" Sep 10 21:17:36.900979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:36.900942 2585 scope.go:117] "RemoveContainer" containerID="27a5ce6de107e8bbe6892cacfadb30c02cf441611d0af13ab5418b5effa6fbac" Sep 10 21:17:49.065002 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:49.064966 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd"] Sep 10 21:17:49.070655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:49.070627 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b5fd067-5rprd"] Sep 10 21:17:50.818188 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:17:50.818152 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34446a73-78f1-4f18-a0f3-06befece1803" path="/var/lib/kubelet/pods/34446a73-78f1-4f18-a0f3-06befece1803/volumes" Sep 10 21:18:36.850085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.850056 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:18:36.850555 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.850214 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:18:36.858098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.858076 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:18:36.858200 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.858076 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:18:36.924327 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.924297 2585 scope.go:117] "RemoveContainer" containerID="df4236aaf132bd902ad80348b99586e38f36eec03ed5bdb3b8204e57602b5ade" Sep 10 21:18:36.932470 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.932445 2585 scope.go:117] "RemoveContainer" containerID="bc0638f70be727f6403e3a3951ffaeef3410849cfadc16ffdfa321581138e407" Sep 10 21:18:36.939976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:36.939959 2585 scope.go:117] "RemoveContainer" containerID="6d11610eaed61d86a656a96c8ffd7b91a65eb004aef7bc5a5d469a2b3ff7b48e" Sep 10 21:18:48.135862 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:48.135829 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:18:48.136341 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:48.136040 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" containerID="cri-o://aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53" gracePeriod=30 Sep 10 21:18:53.231713 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.231678 2585 patch_prober.go:28] interesting pod/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 21:18:53.232119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.231738 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 21:18:53.403029 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.402951 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:18:53.482102 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482074 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482121 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482143 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482227 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482256 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482291 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482312 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482370 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:18:53.482464 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482424 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:18:53.482704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482515 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482558 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4fhn\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn\") pod \"7efad980-8873-4cb6-a600-f3de36007995\" (UID: \"7efad980-8873-4cb6-a600-f3de36007995\") " Sep 10 21:18:53.482704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482647 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:18:53.482704 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482656 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data" (OuterVolumeSpecName: "istio-data") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:18:53.482945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.482920 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:18:53.483044 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.483008 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.483044 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.483031 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.483161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.483044 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.483161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.483059 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.484793 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.484753 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:18:53.484793 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.484763 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token" (OuterVolumeSpecName: "istio-token") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:18:53.485016 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.484985 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn" (OuterVolumeSpecName: "kube-api-access-n4fhn") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "kube-api-access-n4fhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:18:53.485135 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.485118 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7efad980-8873-4cb6-a600-f3de36007995" (UID: "7efad980-8873-4cb6-a600-f3de36007995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:18:53.583572 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.583538 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7efad980-8873-4cb6-a600-f3de36007995-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.583572 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.583569 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7efad980-8873-4cb6-a600-f3de36007995-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.583784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.583579 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4fhn\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-kube-api-access-n4fhn\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.583784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.583589 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7efad980-8873-4cb6-a600-f3de36007995-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:53.583784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:53.583597 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7efad980-8873-4cb6-a600-f3de36007995-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:18:54.247402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.247370 2585 generic.go:356] "Generic (PLEG): container finished" podID="7efad980-8873-4cb6-a600-f3de36007995" containerID="aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53" exitCode=0 Sep 10 21:18:54.247820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.247417 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" Sep 10 21:18:54.247820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.247457 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerDied","Data":"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53"} Sep 10 21:18:54.247820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.247514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5" event={"ID":"7efad980-8873-4cb6-a600-f3de36007995","Type":"ContainerDied","Data":"b4e1cfa29c71b72d88ccf8209cadbff661dd1c91f1185102fc80aed0ed7c3bf1"} Sep 10 21:18:54.247820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.247537 2585 scope.go:117] "RemoveContainer" containerID="aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53" Sep 10 21:18:54.260692 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.260542 2585 scope.go:117] "RemoveContainer" containerID="6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805" Sep 10 21:18:54.268945 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.268924 2585 scope.go:117] "RemoveContainer" containerID="aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53" Sep 10 21:18:54.269243 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:18:54.269216 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53\": container with ID starting with aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53 not found: ID does not exist" containerID="aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53" Sep 10 21:18:54.269309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.269250 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53"} err="failed to get container status \"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53\": rpc error: code = NotFound desc = could not find container \"aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53\": container with ID starting with aa5289a9ba60df8c5d818db5c75035016e800e6eaad27ef99520ce365caafc53 not found: ID does not exist" Sep 10 21:18:54.269309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.269268 2585 scope.go:117] "RemoveContainer" containerID="6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805" Sep 10 21:18:54.269553 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:18:54.269529 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805\": container with ID starting with 6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805 not found: ID does not exist" containerID="6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805" Sep 10 21:18:54.269605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.269560 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805"} err="failed to get container status \"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805\": rpc error: code = NotFound desc = could not find container \"6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805\": container with ID starting with 6ab6316a737369b4f4c7340c00dabd93bd3400cdebe2ac29030a2784a8bab805 not found: ID does not exist" Sep 10 21:18:54.274360 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.274336 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:18:54.279742 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.279724 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8b5fd067-openshift-default-6f8857cf96-b6zf5"] Sep 10 21:18:54.819721 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:18:54.819692 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7efad980-8873-4cb6-a600-f3de36007995" path="/var/lib/kubelet/pods/7efad980-8873-4cb6-a600-f3de36007995/volumes" Sep 10 21:19:36.966445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:36.966375 2585 scope.go:117] "RemoveContainer" containerID="7bae7dc8e6498a829055728fa15dc99b7100da6f08331572eba9a695f40970b3" Sep 10 21:19:46.361371 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361337 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361751 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361764 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361784 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34446a73-78f1-4f18-a0f3-06befece1803" containerName="revoke-keys" Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361789 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="34446a73-78f1-4f18-a0f3-06befece1803" containerName="revoke-keys" Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361799 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361804 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361858 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361867 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efad980-8873-4cb6-a600-f3de36007995" containerName="istio-proxy" Sep 10 21:19:46.361994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.361875 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="34446a73-78f1-4f18-a0f3-06befece1803" containerName="revoke-keys" Sep 10 21:19:46.366430 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.366410 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.380923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.380899 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5a30bd-openshift-default-dockercfg-lqxff\"" Sep 10 21:19:46.425062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.425037 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:19:46.481501 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481501 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481501 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x44\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481664 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481689 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481705 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.481933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.481806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.582803 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.582770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.582994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.582852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.582994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.582872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x44\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.582994 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.582898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583062 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583176 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583112 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583387 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583387 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583212 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583387 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583387 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583370 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583601 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583553 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.583601 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.583595 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.585340 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.585312 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.585523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.585505 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.593394 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.593369 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x44\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.595029 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.594998 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token\") pod \"e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.676644 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.676557 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:46.958178 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:19:46.958149 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24512b1_6fbb_4e1e_a8bc_3f77c81c4adc.slice/crio-e9f8ff6d123fad89151cdb98a3c2dd5720250b01b5d25d8e6452092b9ec1ece1 WatchSource:0}: Error finding container e9f8ff6d123fad89151cdb98a3c2dd5720250b01b5d25d8e6452092b9ec1ece1: Status 404 returned error can't find the container with id e9f8ff6d123fad89151cdb98a3c2dd5720250b01b5d25d8e6452092b9ec1ece1 Sep 10 21:19:46.958984 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.958958 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:19:46.960716 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.960687 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:19:46.960818 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.960749 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:19:46.960818 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:46.960780 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:19:47.536336 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:47.536301 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerStarted","Data":"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916"} Sep 10 21:19:47.536794 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:47.536352 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerStarted","Data":"e9f8ff6d123fad89151cdb98a3c2dd5720250b01b5d25d8e6452092b9ec1ece1"} Sep 10 21:19:47.580221 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:47.580177 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podStartSLOduration=1.580163884 podStartE2EDuration="1.580163884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:47.573774657 +0000 UTC m=+1271.282069062" watchObservedRunningTime="2026-09-10 21:19:47.580163884 +0000 UTC m=+1271.288458269" Sep 10 21:19:47.677277 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:47.677248 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:48.682836 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:48.682801 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:49.547325 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:49.547297 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:19:49.548680 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:19:49.548661 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:20:11.665709 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.665679 2585 generic.go:356] "Generic (PLEG): container finished" podID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerID="1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916" exitCode=0 Sep 10 21:20:11.666187 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.665751 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerDied","Data":"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916"} Sep 10 21:20:11.666187 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.666180 2585 scope.go:117] "RemoveContainer" containerID="1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916" Sep 10 21:20:11.667316 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.667275 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:11.667377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.667357 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:11.667413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:11.667389 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:12.672820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:12.672780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerStarted","Data":"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e"} Sep 10 21:20:12.677891 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:12.677853 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:20:12.678157 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:12.678129 2585 patch_prober.go:28] interesting pod/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 21:20:12.678232 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:12.678202 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 21:20:13.677820 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:13.677776 2585 patch_prober.go:28] interesting pod/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 21:20:13.678332 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:13.677852 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 21:20:14.677234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:14.677194 2585 patch_prober.go:28] interesting pod/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 21:20:14.677450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:14.677264 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 21:20:15.321634 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.321590 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:20:15.325323 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.325302 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.328649 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.328603 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a22115-openshift-default-dockercfg-5sf7v\"" Sep 10 21:20:15.353790 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.353762 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:20:15.452484 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.452445 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.452853 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.452816 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453082 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xxw\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453240 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453226 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453386 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453531 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453531 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453739 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.453739 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.453598 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554561 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554524 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554753 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554577 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xxw\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554753 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554753 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554862 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.554959 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.554938 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555105 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555105 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555058 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555105 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555267 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555112 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555267 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555393 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555302 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.555726 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.555703 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.556875 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.556855 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.557488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.557468 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.567249 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.567224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.567559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.567540 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xxw\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw\") pod \"e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.636011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.635941 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:15.677447 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.677413 2585 patch_prober.go:28] interesting pod/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 21:20:15.677717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.677683 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 21:20:15.961377 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.961353 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:20:15.963467 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:20:15.963432 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea206f26_73b1_43ef_9b66_d6c3110b5dd7.slice/crio-f3b88d6606f2653d576e048da74c694f74cdef6a61dea1488104c18c9cb7a994 WatchSource:0}: Error finding container f3b88d6606f2653d576e048da74c694f74cdef6a61dea1488104c18c9cb7a994: Status 404 returned error can't find the container with id f3b88d6606f2653d576e048da74c694f74cdef6a61dea1488104c18c9cb7a994 Sep 10 21:20:15.965855 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.965823 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:15.965924 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.965887 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:15.965924 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:15.965915 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:16.677331 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:16.677288 2585 patch_prober.go:28] interesting pod/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 21:20:16.677859 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:16.677351 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 21:20:16.711998 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:16.711965 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerStarted","Data":"52284a26f3dd3876607400540aed7c2ead2c4d7a46f7316344751328d4fc3375"} Sep 10 21:20:16.712182 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:16.712017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerStarted","Data":"f3b88d6606f2653d576e048da74c694f74cdef6a61dea1488104c18c9cb7a994"} Sep 10 21:20:16.741086 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:16.741035 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podStartSLOduration=1.7410175319999999 podStartE2EDuration="1.741017532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:16.739119939 +0000 UTC m=+1300.447414326" watchObservedRunningTime="2026-09-10 21:20:16.741017532 +0000 UTC m=+1300.449311918" Sep 10 21:20:17.636304 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.636262 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:17.643027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.642991 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:17.726251 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.726213 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:17.727135 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.727105 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:17.871881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.871850 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:20:17.872405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.872375 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:20:17.873682 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:17.873657 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:20:36.990341 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:36.990307 2585 scope.go:117] "RemoveContainer" containerID="7b3a30500bfebb3886ba6e67fef890ca5b6077161c867a63271d846066ced6de" Sep 10 21:20:40.860350 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.860315 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerID="52284a26f3dd3876607400540aed7c2ead2c4d7a46f7316344751328d4fc3375" exitCode=0 Sep 10 21:20:40.860796 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.860387 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerDied","Data":"52284a26f3dd3876607400540aed7c2ead2c4d7a46f7316344751328d4fc3375"} Sep 10 21:20:40.860843 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.860824 2585 scope.go:117] "RemoveContainer" containerID="52284a26f3dd3876607400540aed7c2ead2c4d7a46f7316344751328d4fc3375" Sep 10 21:20:40.861933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.861903 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:40.861991 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.861972 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:40.862027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:40.862003 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:20:41.868030 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:41.867994 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerStarted","Data":"1ed8a45fd5e32ce1787a470e2562d195349cc50e70ac42912999770d8123aa45"} Sep 10 21:20:42.636694 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:42.636660 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:42.636882 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:42.636859 2585 patch_prober.go:28] interesting pod/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 10 21:20:42.636936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:42.636917 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 10 21:20:43.636413 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:43.636367 2585 patch_prober.go:28] interesting pod/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 10 21:20:43.636808 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:43.636436 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 10 21:20:44.636663 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:44.636602 2585 patch_prober.go:28] interesting pod/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 10 21:20:44.637153 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:44.636688 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 10 21:20:45.636266 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:45.636232 2585 patch_prober.go:28] interesting pod/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 10 21:20:45.636447 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:45.636321 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 10 21:20:46.577324 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.577276 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9"] Sep 10 21:20:46.581697 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.581670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:46.584297 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.584276 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:20:46.584700 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.584683 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:20:46.596476 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.596454 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9"] Sep 10 21:20:46.636587 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.636558 2585 patch_prober.go:28] interesting pod/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 10 21:20:46.636784 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.636606 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 10 21:20:46.662904 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.662877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9\" (UID: \"6c8a0831-90e0-4880-9ee7-547b67df878d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:46.764523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.764488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9\" (UID: \"6c8a0831-90e0-4880-9ee7-547b67df878d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:46.765223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.765202 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9\" (UID: \"6c8a0831-90e0-4880-9ee7-547b67df878d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:46.901685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:46.901570 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:47.218695 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.218664 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9"] Sep 10 21:20:47.219741 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:20:47.219716 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a0831_90e0_4880_9ee7_547b67df878d.slice/crio-88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf WatchSource:0}: Error finding container 88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf: Status 404 returned error can't find the container with id 88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf Sep 10 21:20:47.641030 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.640998 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:47.641505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.641389 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:47.642197 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.642177 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:20:47.914190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.914119 2585 generic.go:356] "Generic (PLEG): container finished" podID="6c8a0831-90e0-4880-9ee7-547b67df878d" containerID="f5a4928f5715816bc11a51af8a2315620fe8c079826c15b3c0aa05b46e3ebe1b" exitCode=0 Sep 10 21:20:47.914190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.914162 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" event={"ID":"6c8a0831-90e0-4880-9ee7-547b67df878d","Type":"ContainerDied","Data":"f5a4928f5715816bc11a51af8a2315620fe8c079826c15b3c0aa05b46e3ebe1b"} Sep 10 21:20:47.914347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:47.914217 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" event={"ID":"6c8a0831-90e0-4880-9ee7-547b67df878d","Type":"ContainerStarted","Data":"88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf"} Sep 10 21:20:49.054306 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.054281 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:49.187046 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.186968 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca\") pod \"6c8a0831-90e0-4880-9ee7-547b67df878d\" (UID: \"6c8a0831-90e0-4880-9ee7-547b67df878d\") " Sep 10 21:20:49.187361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.187330 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c8a0831-90e0-4880-9ee7-547b67df878d" (UID: "6c8a0831-90e0-4880-9ee7-547b67df878d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:20:49.288043 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.288015 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8a0831-90e0-4880-9ee7-547b67df878d-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:20:49.926668 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.926637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" event={"ID":"6c8a0831-90e0-4880-9ee7-547b67df878d","Type":"ContainerDied","Data":"88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf"} Sep 10 21:20:49.926832 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.926682 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9" Sep 10 21:20:49.926832 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:20:49.926685 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c13fc4db2ef3b7fe14d9ff4b84b7f397cd018969757563b00fb03f2fd4cddf" Sep 10 21:21:04.412268 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:04.412175 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9"] Sep 10 21:21:04.415684 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:04.415643 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5a30bd-j2lf9"] Sep 10 21:21:04.818022 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:04.817989 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8a0831-90e0-4880-9ee7-547b67df878d" path="/var/lib/kubelet/pods/6c8a0831-90e0-4880-9ee7-547b67df878d/volumes" Sep 10 21:21:11.707589 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:11.707557 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:21:11.708213 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:11.707788 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" containerID="cri-o://e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e" gracePeriod=30 Sep 10 21:21:12.076887 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.076855 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4"] Sep 10 21:21:12.077253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.077241 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8a0831-90e0-4880-9ee7-547b67df878d" containerName="revoke-keys" Sep 10 21:21:12.077308 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.077255 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8a0831-90e0-4880-9ee7-547b67df878d" containerName="revoke-keys" Sep 10 21:21:12.077348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.077333 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8a0831-90e0-4880-9ee7-547b67df878d" containerName="revoke-keys" Sep 10 21:21:12.080446 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.080429 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:12.083166 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.083145 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:21:12.083290 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.083233 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:21:12.094662 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.094637 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4"] Sep 10 21:21:12.212961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.212932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a22115-p4sz4\" (UID: \"a3923c20-c736-4827-b674-3979722493d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:12.314215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.314171 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a22115-p4sz4\" (UID: \"a3923c20-c736-4827-b674-3979722493d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:12.314840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.314820 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a22115-p4sz4\" (UID: \"a3923c20-c736-4827-b674-3979722493d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:12.390348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.390285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:12.662767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:12.662740 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4"] Sep 10 21:21:12.663789 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:21:12.663761 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3923c20_c736_4827_b674_3979722493d1.slice/crio-73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a WatchSource:0}: Error finding container 73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a: Status 404 returned error can't find the container with id 73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a Sep 10 21:21:13.048428 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:13.048400 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" event={"ID":"a3923c20-c736-4827-b674-3979722493d1","Type":"ContainerStarted","Data":"30d3410cd96b1a4bd132d350944c750ace8d0a9da412956bdf71baa42487eedb"} Sep 10 21:21:13.048768 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:13.048450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" event={"ID":"a3923c20-c736-4827-b674-3979722493d1","Type":"ContainerStarted","Data":"73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a"} Sep 10 21:21:13.071025 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:13.070975 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" podStartSLOduration=1.070960823 podStartE2EDuration="1.070960823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:13.069258444 +0000 UTC m=+1356.777552838" watchObservedRunningTime="2026-09-10 21:21:13.070960823 +0000 UTC m=+1356.779255208" Sep 10 21:21:14.054781 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:14.054753 2585 generic.go:356] "Generic (PLEG): container finished" podID="a3923c20-c736-4827-b674-3979722493d1" containerID="30d3410cd96b1a4bd132d350944c750ace8d0a9da412956bdf71baa42487eedb" exitCode=0 Sep 10 21:21:14.055161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:14.054838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" event={"ID":"a3923c20-c736-4827-b674-3979722493d1","Type":"ContainerDied","Data":"30d3410cd96b1a4bd132d350944c750ace8d0a9da412956bdf71baa42487eedb"} Sep 10 21:21:15.191375 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:15.191352 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:15.246576 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:15.246544 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca\") pod \"a3923c20-c736-4827-b674-3979722493d1\" (UID: \"a3923c20-c736-4827-b674-3979722493d1\") " Sep 10 21:21:15.246894 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:15.246868 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3923c20-c736-4827-b674-3979722493d1" (UID: "a3923c20-c736-4827-b674-3979722493d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:21:15.347692 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:15.347605 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3923c20-c736-4827-b674-3979722493d1-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:16.066520 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:16.066488 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" Sep 10 21:21:16.066716 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:16.066527 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4" event={"ID":"a3923c20-c736-4827-b674-3979722493d1","Type":"ContainerDied","Data":"73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a"} Sep 10 21:21:16.066716 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:16.066573 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c8d678134043ce96619eabe5d334a4f887eb4bfcbdd7801dadc99659a1262a" Sep 10 21:21:16.955675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:16.955652 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:21:17.061874 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061852 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061885 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061911 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061933 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24x44\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061954 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.061984 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062055 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062053 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062375 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062170 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data" (OuterVolumeSpecName: "istio-data") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:17.062375 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062179 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062375 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062257 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert\") pod \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\" (UID: \"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc\") " Sep 10 21:21:17.062513 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062359 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:17.062572 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062513 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:17.062711 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062685 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:17.062836 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062814 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:21:17.062884 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062860 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.062884 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062879 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.062964 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062894 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.062964 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.062907 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.064186 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.064160 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token" (OuterVolumeSpecName: "istio-token") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:21:17.064298 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.064209 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:21:17.064476 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.064459 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:17.064541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.064514 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44" (OuterVolumeSpecName: "kube-api-access-24x44") pod "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" (UID: "b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc"). InnerVolumeSpecName "kube-api-access-24x44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:21:17.074743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.074716 2585 generic.go:356] "Generic (PLEG): container finished" podID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerID="e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e" exitCode=0 Sep 10 21:21:17.074858 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.074768 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" Sep 10 21:21:17.074858 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.074799 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerDied","Data":"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e"} Sep 10 21:21:17.074976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.074855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj" event={"ID":"b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc","Type":"ContainerDied","Data":"e9f8ff6d123fad89151cdb98a3c2dd5720250b01b5d25d8e6452092b9ec1ece1"} Sep 10 21:21:17.074976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.074885 2585 scope.go:117] "RemoveContainer" containerID="e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e" Sep 10 21:21:17.084541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.084524 2585 scope.go:117] "RemoveContainer" containerID="1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916" Sep 10 21:21:17.092435 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.092416 2585 scope.go:117] "RemoveContainer" containerID="e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e" Sep 10 21:21:17.092680 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:21:17.092663 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e\": container with ID starting with e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e not found: ID does not exist" containerID="e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e" Sep 10 21:21:17.092731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.092687 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e"} err="failed to get container status \"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e\": rpc error: code = NotFound desc = could not find container \"e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e\": container with ID starting with e762095a32e66ba1b319c235ec86f192d7d4de467ecdd20533f5c1a5e732151e not found: ID does not exist" Sep 10 21:21:17.092731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.092702 2585 scope.go:117] "RemoveContainer" containerID="1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916" Sep 10 21:21:17.092938 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:21:17.092923 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916\": container with ID starting with 1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916 not found: ID does not exist" containerID="1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916" Sep 10 21:21:17.092975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.092942 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916"} err="failed to get container status \"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916\": rpc error: code = NotFound desc = could not find container \"1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916\": container with ID starting with 1672cc47c39f6f82d90b720a9fc07294b627ebaabe79bf48eaaf46b0cb611916 not found: ID does not exist" Sep 10 21:21:17.111789 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.111768 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:21:17.121336 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.121316 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5a30bd-openshift-default-6f6bb7bc46-zmfdj"] Sep 10 21:21:17.163354 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.163331 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.163354 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.163354 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.163482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.163367 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.163482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.163379 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x44\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-kube-api-access-24x44\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:17.163482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:17.163393 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:18.818291 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:18.818256 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" path="/var/lib/kubelet/pods/b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc/volumes" Sep 10 21:21:28.333836 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:28.333802 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4"] Sep 10 21:21:28.339327 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:28.339295 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a22115-p4sz4"] Sep 10 21:21:28.818210 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:28.818168 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3923c20-c736-4827-b674-3979722493d1" path="/var/lib/kubelet/pods/a3923c20-c736-4827-b674-3979722493d1/volumes" Sep 10 21:21:36.033253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.033212 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:21:36.033743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.033493 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" containerID="cri-o://1ed8a45fd5e32ce1787a470e2562d195349cc50e70ac42912999770d8123aa45" gracePeriod=30 Sep 10 21:21:36.446996 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.446918 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd"] Sep 10 21:21:36.447356 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447341 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:36.447402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447360 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:36.447402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447369 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:36.447402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447374 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:36.447502 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447409 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3923c20-c736-4827-b674-3979722493d1" containerName="revoke-keys" Sep 10 21:21:36.447502 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447415 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3923c20-c736-4827-b674-3979722493d1" containerName="revoke-keys" Sep 10 21:21:36.447502 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447479 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:36.447502 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.447489 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3923c20-c736-4827-b674-3979722493d1" containerName="revoke-keys" Sep 10 21:21:36.450582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.450564 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:36.455184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.455161 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:21:36.458214 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.458191 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:21:36.512855 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.512825 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd"] Sep 10 21:21:36.534292 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.534264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd\" (UID: \"508f7bab-8282-4826-b36e-bf88a2762159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:36.635402 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.635374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd\" (UID: \"508f7bab-8282-4826-b36e-bf88a2762159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:36.636020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.635999 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd\" (UID: \"508f7bab-8282-4826-b36e-bf88a2762159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:36.769153 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.766230 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:21:36.774755 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:36.774727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:37.014285 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:37.014256 2585 scope.go:117] "RemoveContainer" containerID="419c531c56f6bf9796a2e2453efe08ddca745d566e35e67503ce8dc919bd8f87" Sep 10 21:21:37.022880 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:37.022858 2585 scope.go:117] "RemoveContainer" containerID="ab4ce81a2d74017bb2e130e3240a543fef3c4fbe5c44304d722ae90bb77f23a4" Sep 10 21:21:37.049840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:37.049790 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd"] Sep 10 21:21:37.052017 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:21:37.051994 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f7bab_8282_4826_b36e_bf88a2762159.slice/crio-3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a WatchSource:0}: Error finding container 3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a: Status 404 returned error can't find the container with id 3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a Sep 10 21:21:37.182426 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:37.182394 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" event={"ID":"508f7bab-8282-4826-b36e-bf88a2762159","Type":"ContainerStarted","Data":"3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a"} Sep 10 21:21:38.189084 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:38.188996 2585 generic.go:356] "Generic (PLEG): container finished" podID="508f7bab-8282-4826-b36e-bf88a2762159" containerID="ec8c914c66be9e0276ed87bb215148085f261a9c1a931842072de11ac7738e03" exitCode=0 Sep 10 21:21:38.189084 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:38.189075 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" event={"ID":"508f7bab-8282-4826-b36e-bf88a2762159","Type":"ContainerDied","Data":"ec8c914c66be9e0276ed87bb215148085f261a9c1a931842072de11ac7738e03"} Sep 10 21:21:39.325606 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:39.325582 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:39.462699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:39.462602 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca\") pod \"508f7bab-8282-4826-b36e-bf88a2762159\" (UID: \"508f7bab-8282-4826-b36e-bf88a2762159\") " Sep 10 21:21:39.462976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:39.462951 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "508f7bab-8282-4826-b36e-bf88a2762159" (UID: "508f7bab-8282-4826-b36e-bf88a2762159"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:21:39.564003 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:39.563974 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508f7bab-8282-4826-b36e-bf88a2762159-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:40.200451 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:40.200422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" Sep 10 21:21:40.200666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:40.200470 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd" event={"ID":"508f7bab-8282-4826-b36e-bf88a2762159","Type":"ContainerDied","Data":"3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a"} Sep 10 21:21:40.200666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:40.200518 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aa7fe90cde759c7089cf93713ef4e461900e0b0abe7fce95dac786765d0f33a" Sep 10 21:21:41.209841 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.209806 2585 generic.go:356] "Generic (PLEG): container finished" podID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerID="1ed8a45fd5e32ce1787a470e2562d195349cc50e70ac42912999770d8123aa45" exitCode=0 Sep 10 21:21:41.210260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.209919 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerDied","Data":"1ed8a45fd5e32ce1787a470e2562d195349cc50e70ac42912999770d8123aa45"} Sep 10 21:21:41.210260 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.209977 2585 scope.go:117] "RemoveContainer" containerID="52284a26f3dd3876607400540aed7c2ead2c4d7a46f7316344751328d4fc3375" Sep 10 21:21:41.278681 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.278660 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:21:41.381780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381716 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381780 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381751 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381787 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381832 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381862 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381880 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381907 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7xxw\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.381978 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.381944 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.382295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy\") pod \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\" (UID: \"ea206f26-73b1-43ef-9b66-d6c3110b5dd7\") " Sep 10 21:21:41.382295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382063 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:41.382295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382176 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:41.382295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382092 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382302 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382394 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382458 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382471 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382484 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.382517 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.382498 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.384094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.384065 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw" (OuterVolumeSpecName: "kube-api-access-b7xxw") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "kube-api-access-b7xxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:21:41.384223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.384160 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:21:41.384458 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.384441 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:21:41.384522 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.384463 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea206f26-73b1-43ef-9b66-d6c3110b5dd7" (UID: "ea206f26-73b1-43ef-9b66-d6c3110b5dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:21:41.483081 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.483052 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.483081 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.483076 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.483234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.483090 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xxw\" (UniqueName: \"kubernetes.io/projected/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-kube-api-access-b7xxw\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.483234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.483104 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:41.483234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:41.483115 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea206f26-73b1-43ef-9b66-d6c3110b5dd7-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:42.217317 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.217277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" event={"ID":"ea206f26-73b1-43ef-9b66-d6c3110b5dd7","Type":"ContainerDied","Data":"f3b88d6606f2653d576e048da74c694f74cdef6a61dea1488104c18c9cb7a994"} Sep 10 21:21:42.217771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.217340 2585 scope.go:117] "RemoveContainer" containerID="1ed8a45fd5e32ce1787a470e2562d195349cc50e70ac42912999770d8123aa45" Sep 10 21:21:42.217771 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.217300 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm" Sep 10 21:21:42.241814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.241787 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:21:42.245539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.245510 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a22115-openshift-default-74fb7c946b-j79fm"] Sep 10 21:21:42.817394 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:42.817365 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" path="/var/lib/kubelet/pods/ea206f26-73b1-43ef-9b66-d6c3110b5dd7/volumes" Sep 10 21:21:47.446250 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:47.446214 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd"] Sep 10 21:21:47.449490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:47.449466 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-290e03-n5sjd"] Sep 10 21:21:48.817655 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:48.817603 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508f7bab-8282-4826-b36e-bf88a2762159" path="/var/lib/kubelet/pods/508f7bab-8282-4826-b36e-bf88a2762159/volumes" Sep 10 21:21:54.755885 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:54.755853 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:21:54.756295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:54.756061 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" containerID="cri-o://66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589" gracePeriod=30 Sep 10 21:21:55.035625 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.035574 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4"] Sep 10 21:21:55.036035 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036019 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508f7bab-8282-4826-b36e-bf88a2762159" containerName="revoke-keys" Sep 10 21:21:55.036092 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036038 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="508f7bab-8282-4826-b36e-bf88a2762159" containerName="revoke-keys" Sep 10 21:21:55.036092 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036050 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:21:55.036092 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036056 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:21:55.036092 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036065 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:21:55.036092 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036071 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:21:55.036265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036137 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24512b1-6fbb-4e1e-a8bc-3f77c81c4adc" containerName="istio-proxy" Sep 10 21:21:55.036265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036147 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:21:55.036265 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.036156 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="508f7bab-8282-4826-b36e-bf88a2762159" containerName="revoke-keys" Sep 10 21:21:55.040839 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.040823 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:55.044322 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.044300 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:21:55.044437 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.044320 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:21:55.048293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.048268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4"] Sep 10 21:21:55.205930 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.205897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4\" (UID: \"1f620396-dff4-4ce0-8c3f-18427bb64043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:55.307039 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.306944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4\" (UID: \"1f620396-dff4-4ce0-8c3f-18427bb64043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:55.307552 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.307532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4\" (UID: \"1f620396-dff4-4ce0-8c3f-18427bb64043\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:55.351511 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.351486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:55.627903 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:55.627881 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4"] Sep 10 21:21:55.630197 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:21:55.630173 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f620396_dff4_4ce0_8c3f_18427bb64043.slice/crio-cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1 WatchSource:0}: Error finding container cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1: Status 404 returned error can't find the container with id cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1 Sep 10 21:21:56.294292 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:56.294259 2585 generic.go:356] "Generic (PLEG): container finished" podID="1f620396-dff4-4ce0-8c3f-18427bb64043" containerID="c363bc12d57d6689bcbae47e466e0a8edde155215bc84e8db7c3c03fe0711dbb" exitCode=0 Sep 10 21:21:56.294635 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:56.294344 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" event={"ID":"1f620396-dff4-4ce0-8c3f-18427bb64043","Type":"ContainerDied","Data":"c363bc12d57d6689bcbae47e466e0a8edde155215bc84e8db7c3c03fe0711dbb"} Sep 10 21:21:56.294635 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:56.294389 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" event={"ID":"1f620396-dff4-4ce0-8c3f-18427bb64043","Type":"ContainerStarted","Data":"cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1"} Sep 10 21:21:57.429389 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:57.429364 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:57.525618 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:57.525577 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca\") pod \"1f620396-dff4-4ce0-8c3f-18427bb64043\" (UID: \"1f620396-dff4-4ce0-8c3f-18427bb64043\") " Sep 10 21:21:57.525938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:57.525916 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f620396-dff4-4ce0-8c3f-18427bb64043" (UID: "1f620396-dff4-4ce0-8c3f-18427bb64043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:21:57.627189 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:57.627118 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f620396-dff4-4ce0-8c3f-18427bb64043-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:21:58.234341 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:58.234303 2585 patch_prober.go:28] interesting pod/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 21:21:58.234510 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:58.234363 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:21:58.305774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:58.305742 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" Sep 10 21:21:58.305774 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:58.305754 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4" event={"ID":"1f620396-dff4-4ce0-8c3f-18427bb64043","Type":"ContainerDied","Data":"cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1"} Sep 10 21:21:58.305983 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:21:58.305804 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc4afcf24105e1a2e496c906ed0be15af5897b70751cd5bfd62a24a4045d97e1" Sep 10 21:22:00.018173 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.018106 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:22:00.056067 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056041 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056078 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056110 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx4pz\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056136 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056153 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056218 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056192 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056228 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056328 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo\") pod \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\" (UID: \"0955c7ed-ef80-44dc-b859-d48bcdc601d8\") " Sep 10 21:22:00.056675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056534 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:00.056675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056553 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:00.056675 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056630 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:00.056828 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056800 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.056828 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056820 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.056934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056836 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.056934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056860 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:00.056934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.056892 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:22:00.058485 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.058461 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:22:00.058801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.058779 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:00.058969 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.058804 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz" (OuterVolumeSpecName: "kube-api-access-rx4pz") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "kube-api-access-rx4pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:00.059093 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.059068 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0955c7ed-ef80-44dc-b859-d48bcdc601d8" (UID: "0955c7ed-ef80-44dc-b859-d48bcdc601d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:00.157600 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157571 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.157600 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157594 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.157600 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157603 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.157600 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157624 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.157862 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157636 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.157862 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.157648 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4pz\" (UniqueName: \"kubernetes.io/projected/0955c7ed-ef80-44dc-b859-d48bcdc601d8-kube-api-access-rx4pz\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.317923 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.317894 2585 generic.go:356] "Generic (PLEG): container finished" podID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerID="66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589" exitCode=0 Sep 10 21:22:00.318112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.317943 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" Sep 10 21:22:00.318112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.317980 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerDied","Data":"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589"} Sep 10 21:22:00.318112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.318055 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2" event={"ID":"0955c7ed-ef80-44dc-b859-d48bcdc601d8","Type":"ContainerDied","Data":"1ac2a08aa27f767bf15c5ba3d87e60566de5a8d8fd515b0578738de247dbd78f"} Sep 10 21:22:00.318112 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.318078 2585 scope.go:117] "RemoveContainer" containerID="66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589" Sep 10 21:22:00.327639 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.327603 2585 scope.go:117] "RemoveContainer" containerID="e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649" Sep 10 21:22:00.335593 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.335577 2585 scope.go:117] "RemoveContainer" containerID="66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589" Sep 10 21:22:00.335875 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:22:00.335857 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589\": container with ID starting with 66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589 not found: ID does not exist" containerID="66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589" Sep 10 21:22:00.335935 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.335885 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589"} err="failed to get container status \"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589\": rpc error: code = NotFound desc = could not find container \"66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589\": container with ID starting with 66efe1a2ca89e36fdca696dfab447bb81881f9848ab709d3497e713da9ab0589 not found: ID does not exist" Sep 10 21:22:00.335935 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.335901 2585 scope.go:117] "RemoveContainer" containerID="e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649" Sep 10 21:22:00.336166 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:22:00.336151 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649\": container with ID starting with e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649 not found: ID does not exist" containerID="e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649" Sep 10 21:22:00.336210 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.336170 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649"} err="failed to get container status \"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649\": rpc error: code = NotFound desc = could not find container \"e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649\": container with ID starting with e9c34940f58acd50638451a4ebb508ba760a07171765daab693a295676067649 not found: ID does not exist" Sep 10 21:22:00.351421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.351396 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:22:00.356230 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.356210 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-290e03-openshift-default-7ccd8ff65b-55jw2"] Sep 10 21:22:00.818215 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:00.818174 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" path="/var/lib/kubelet/pods/0955c7ed-ef80-44dc-b859-d48bcdc601d8/volumes" Sep 10 21:22:05.135169 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135139 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:22:05.135537 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135522 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.135537 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135534 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.135631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135554 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f620396-dff4-4ce0-8c3f-18427bb64043" containerName="revoke-keys" Sep 10 21:22:05.135631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135559 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f620396-dff4-4ce0-8c3f-18427bb64043" containerName="revoke-keys" Sep 10 21:22:05.135631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135567 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.135631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135572 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.135760 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135645 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.135760 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135652 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f620396-dff4-4ce0-8c3f-18427bb64043" containerName="revoke-keys" Sep 10 21:22:05.135760 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135659 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea206f26-73b1-43ef-9b66-d6c3110b5dd7" containerName="istio-proxy" Sep 10 21:22:05.135760 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.135667 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0955c7ed-ef80-44dc-b859-d48bcdc601d8" containerName="istio-proxy" Sep 10 21:22:05.140145 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.140126 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.143182 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.143158 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-84b69f-openshift-default-dockercfg-n2qnw\"" Sep 10 21:22:05.151684 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.151658 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:22:05.204967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.204934 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205130 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.204998 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205130 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205177 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205223 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcdt\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205457 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205304 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.205457 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.205339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcdt\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306655 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306774 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306838 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.306934 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.306887 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.307183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.307030 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.307432 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.307414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.308990 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.308972 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.309296 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.309274 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.317453 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.317405 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcdt\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.317691 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.317676 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token\") pod \"e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.451572 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.451489 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:05.733012 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.732983 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:22:05.734669 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:22:05.734641 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ebc914_92d4_4fff_b945_6412b7ba457d.slice/crio-3a71988a55a8d99be913a2285e54d5a8495cceb5d55bd13749a1bd3412dc2de6 WatchSource:0}: Error finding container 3a71988a55a8d99be913a2285e54d5a8495cceb5d55bd13749a1bd3412dc2de6: Status 404 returned error can't find the container with id 3a71988a55a8d99be913a2285e54d5a8495cceb5d55bd13749a1bd3412dc2de6 Sep 10 21:22:05.736908 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.736874 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:05.737020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.736958 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:05.737020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.737003 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:05.880933 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.880902 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4"] Sep 10 21:22:05.886553 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:05.886525 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f71ac-hhtx4"] Sep 10 21:22:06.353601 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.353564 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerStarted","Data":"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58"} Sep 10 21:22:06.354069 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.353633 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerStarted","Data":"3a71988a55a8d99be913a2285e54d5a8495cceb5d55bd13749a1bd3412dc2de6"} Sep 10 21:22:06.395824 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.395772 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podStartSLOduration=1.395756156 podStartE2EDuration="1.395756156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:06.395186719 +0000 UTC m=+1410.103481105" watchObservedRunningTime="2026-09-10 21:22:06.395756156 +0000 UTC m=+1410.104050542" Sep 10 21:22:06.452090 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.452062 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:06.457020 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.456997 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:06.817731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:06.817702 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f620396-dff4-4ce0-8c3f-18427bb64043" path="/var/lib/kubelet/pods/1f620396-dff4-4ce0-8c3f-18427bb64043/volumes" Sep 10 21:22:07.358575 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:07.358548 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:07.359489 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:07.359474 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:12.419847 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:12.419817 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:22:12.420315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:12.420001 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" containerID="cri-o://7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b" gracePeriod=30 Sep 10 21:22:17.658186 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.658160 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:22:17.730625 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730550 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730734 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730628 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730734 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730678 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730737 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730768 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730888 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730842 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swsm7\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.730888 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730868 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.731028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730912 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.731028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.730942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert\") pod \"4caef289-5318-45d0-952a-26e4e82d47fc\" (UID: \"4caef289-5318-45d0-952a-26e4e82d47fc\") " Sep 10 21:22:17.731028 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731011 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:17.731184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731113 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:17.731240 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731219 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:17.731280 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731236 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:17.731412 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731397 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.731465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731419 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.731465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731434 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.731465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731447 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.731465 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.731430 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:22:17.732833 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.732807 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:17.732957 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.732938 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:22:17.733251 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.733232 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7" (OuterVolumeSpecName: "kube-api-access-swsm7") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "kube-api-access-swsm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:17.733290 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.733243 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4caef289-5318-45d0-952a-26e4e82d47fc" (UID: "4caef289-5318-45d0-952a-26e4e82d47fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:22:17.832432 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.832409 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsm7\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-kube-api-access-swsm7\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.832432 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.832431 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4caef289-5318-45d0-952a-26e4e82d47fc-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.832565 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.832440 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4caef289-5318-45d0-952a-26e4e82d47fc-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.832565 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.832448 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4caef289-5318-45d0-952a-26e4e82d47fc-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:17.832565 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:17.832456 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4caef289-5318-45d0-952a-26e4e82d47fc-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:22:18.421572 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.421526 2585 generic.go:356] "Generic (PLEG): container finished" podID="4caef289-5318-45d0-952a-26e4e82d47fc" containerID="7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b" exitCode=0 Sep 10 21:22:18.421764 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.421587 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" Sep 10 21:22:18.421764 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.421638 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerDied","Data":"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b"} Sep 10 21:22:18.421764 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.421712 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc" event={"ID":"4caef289-5318-45d0-952a-26e4e82d47fc","Type":"ContainerDied","Data":"dd25f631ac6ae81ad33eca54e5e8359c3ce6fce7472f9b865709eb9fc7bd0ed5"} Sep 10 21:22:18.421764 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.421743 2585 scope.go:117] "RemoveContainer" containerID="7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b" Sep 10 21:22:18.431397 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.431379 2585 scope.go:117] "RemoveContainer" containerID="0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5" Sep 10 21:22:18.439433 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.439413 2585 scope.go:117] "RemoveContainer" containerID="7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b" Sep 10 21:22:18.439706 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:22:18.439682 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b\": container with ID starting with 7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b not found: ID does not exist" containerID="7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b" Sep 10 21:22:18.439787 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.439717 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b"} err="failed to get container status \"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b\": rpc error: code = NotFound desc = could not find container \"7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b\": container with ID starting with 7b20949da6e37f151443eb71c977d8fdefae6bdc60bdcb0522dbc9576e302e8b not found: ID does not exist" Sep 10 21:22:18.439787 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.439740 2585 scope.go:117] "RemoveContainer" containerID="0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5" Sep 10 21:22:18.439997 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:22:18.439973 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5\": container with ID starting with 0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5 not found: ID does not exist" containerID="0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5" Sep 10 21:22:18.440095 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.440004 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5"} err="failed to get container status \"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5\": rpc error: code = NotFound desc = could not find container \"0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5\": container with ID starting with 0eed51b61d8c9bc4799c3d6f0ae3bc4220bb6f5f52b3f2f04ff66682f576d6c5 not found: ID does not exist" Sep 10 21:22:18.444075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.444052 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:22:18.450319 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.450295 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f71ac-openshift-default-6d4b89d49d-5mdxc"] Sep 10 21:22:18.817263 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:18.817229 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" path="/var/lib/kubelet/pods/4caef289-5318-45d0-952a-26e4e82d47fc/volumes" Sep 10 21:22:34.170508 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.170421 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:22:34.171075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.171022 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:22:34.171075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.171043 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:22:34.171075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.171068 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:22:34.171075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.171076 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:22:34.171293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.171237 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:22:34.183302 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.183277 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.210059 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.208622 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-868994-openshift-default-dockercfg-v9xf8\"" Sep 10 21:22:34.210788 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.210758 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:22:34.286220 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.286176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.286589 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.286554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.286817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.286798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.286975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.286961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.287098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.287085 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.287208 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.287197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.287317 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.287305 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.287457 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.287444 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfwb\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.287600 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.287587 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.388807 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.388761 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.389519 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.389404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.390150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.390132 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.390688 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.390668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.391315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.391122 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.391736 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.391693 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.391861 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.391842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.391993 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.391977 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.393185 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.392348 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.393310 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.392866 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.393400 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.393081 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.393574 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.393554 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfwb\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.394021 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.394001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.394895 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.394876 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.395363 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.394757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.395496 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.395271 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.406664 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.406645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfwb\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.415184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.415165 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token\") pod \"e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.519234 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.519146 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:34.806226 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.806195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:22:34.809469 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:22:34.809442 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ee9b7_4977_4bc0_8b39_6fee4e939ed4.slice/crio-ea013a72d99089285df44f6af5f7a121c8d785a9d04c57ac1c7dd3c99f7b215f WatchSource:0}: Error finding container ea013a72d99089285df44f6af5f7a121c8d785a9d04c57ac1c7dd3c99f7b215f: Status 404 returned error can't find the container with id ea013a72d99089285df44f6af5f7a121c8d785a9d04c57ac1c7dd3c99f7b215f Sep 10 21:22:34.811684 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.811648 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:34.811808 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.811736 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:34.811808 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:34.811783 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:35.517042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.517000 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerStarted","Data":"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0"} Sep 10 21:22:35.517431 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.517101 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerStarted","Data":"ea013a72d99089285df44f6af5f7a121c8d785a9d04c57ac1c7dd3c99f7b215f"} Sep 10 21:22:35.520236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.520212 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:35.521873 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.521850 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:22:35.521955 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.521903 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:22:35.539948 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:35.539905 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podStartSLOduration=1.539890875 podStartE2EDuration="1.539890875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:35.537365776 +0000 UTC m=+1439.245660162" watchObservedRunningTime="2026-09-10 21:22:35.539890875 +0000 UTC m=+1439.248185261" Sep 10 21:22:36.523335 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:36.523308 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:37.528042 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:37.528007 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:37.529130 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:37.529108 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:38.400754 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:38.400717 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 21:22:38.400952 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:38.400846 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 21:22:39.541053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.541021 2585 generic.go:356] "Generic (PLEG): container finished" podID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerID="767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58" exitCode=0 Sep 10 21:22:39.541429 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.541103 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerDied","Data":"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58"} Sep 10 21:22:39.541650 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.541635 2585 scope.go:117] "RemoveContainer" containerID="767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58" Sep 10 21:22:39.542624 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.542590 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:39.542696 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.542680 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:39.542735 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:39.542712 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:40.549603 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:40.549563 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerStarted","Data":"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15"} Sep 10 21:22:41.451893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:41.451854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:41.452094 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:41.452072 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:41.452142 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:41.452121 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:42.452589 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:42.452553 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:42.453033 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:42.452606 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:43.452010 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:43.451967 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:43.452212 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:43.452042 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:44.452417 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:44.452380 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:44.452823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:44.452436 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:45.452264 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:45.452227 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:45.452477 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:45.452289 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:46.452361 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:46.452323 2585 patch_prober.go:28] interesting pod/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 10 21:22:46.452523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:46.452392 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 10 21:22:47.456053 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:47.456017 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:47.456401 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:47.456370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:47.457098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:47.457077 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:22:56.643073 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.643039 2585 generic.go:356] "Generic (PLEG): container finished" podID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerID="cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0" exitCode=0 Sep 10 21:22:56.643498 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.643116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerDied","Data":"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0"} Sep 10 21:22:56.643498 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.643479 2585 scope.go:117] "RemoveContainer" containerID="cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0" Sep 10 21:22:56.644631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.644591 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:56.644695 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.644676 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:56.644732 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:56.644708 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:22:57.650671 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:57.650637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerStarted","Data":"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa"} Sep 10 21:22:58.519840 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:58.519801 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:22:58.520084 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:58.520057 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:22:58.520168 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:58.520118 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:22:59.519932 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:59.519900 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:22:59.520338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:22:59.519957 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:23:00.519976 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:00.519939 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:23:00.520460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:00.520003 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:23:01.519916 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:01.519873 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:23:01.520101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:01.519935 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:23:02.519631 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:02.519576 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 10 21:23:02.519813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:02.519650 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 10 21:23:03.523398 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:03.523369 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:23:03.523776 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:03.523639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:23:03.524200 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:03.524182 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:23:36.885322 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:36.885291 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:23:36.886256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:36.886230 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:23:36.894187 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:36.894163 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:23:36.894573 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:36.894556 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:23:37.073956 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:37.073928 2585 scope.go:117] "RemoveContainer" containerID="b2428b41f6d4eef225f4412c634c91a6dad17b5ef87771b502ee9c496a0f2c62" Sep 10 21:23:49.353802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.353767 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:23:49.354261 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.354245 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caef289-5318-45d0-952a-26e4e82d47fc" containerName="istio-proxy" Sep 10 21:23:49.357376 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.357358 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.360961 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.360936 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-6e4b80-openshift-default-dockercfg-xxjzm\"" Sep 10 21:23:49.378045 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.378023 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:23:49.497627 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497575 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497636 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzp7\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497812 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497769 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497930 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.497975 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.497970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.498166 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.498050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599045 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.598996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599045 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599162 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzp7\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599260 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599367 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599419 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599490 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599471 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599677 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599719 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.599851 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.599751 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.600179 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.600156 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.601331 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.601307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.601650 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.601631 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.610279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.610215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.612391 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.612368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzp7\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7\") pod \"e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.667384 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.667357 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:49.957011 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.956988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:23:49.959292 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:23:49.959257 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18141be7_30d4_47f1_a8ed_7f0e104316bb.slice/crio-86fdb303aeef08cd26dd809fa227ad9189e83f346ecf10a3e8d69643b1185bbb WatchSource:0}: Error finding container 86fdb303aeef08cd26dd809fa227ad9189e83f346ecf10a3e8d69643b1185bbb: Status 404 returned error can't find the container with id 86fdb303aeef08cd26dd809fa227ad9189e83f346ecf10a3e8d69643b1185bbb Sep 10 21:23:49.961505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.961472 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:23:49.961602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.961541 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:23:49.961602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:49.961577 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:23:50.929867 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:50.929831 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerStarted","Data":"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1"} Sep 10 21:23:50.930366 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:50.929892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerStarted","Data":"86fdb303aeef08cd26dd809fa227ad9189e83f346ecf10a3e8d69643b1185bbb"} Sep 10 21:23:50.954579 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:50.954521 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podStartSLOduration=1.954505451 podStartE2EDuration="1.954505451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:50.951964038 +0000 UTC m=+1514.660258435" watchObservedRunningTime="2026-09-10 21:23:50.954505451 +0000 UTC m=+1514.662799836" Sep 10 21:23:51.667712 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:51.667676 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:51.672488 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:51.672468 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:51.935346 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:51.935269 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:23:51.936106 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:23:51.936089 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:18.622414 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.622373 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6"] Sep 10 21:24:18.625993 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.625974 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:18.633843 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.633818 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:24:18.633947 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.633898 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:24:18.659178 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.659149 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6"] Sep 10 21:24:18.669802 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.669770 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6\" (UID: \"467dbc45-e1ae-4e04-9435-fbd889ad539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:18.771111 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.771075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6\" (UID: \"467dbc45-e1ae-4e04-9435-fbd889ad539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:18.771759 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.771736 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6\" (UID: \"467dbc45-e1ae-4e04-9435-fbd889ad539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:18.936498 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:18.936417 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:19.234003 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:19.233979 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6"] Sep 10 21:24:19.236262 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:24:19.236235 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467dbc45_e1ae_4e04_9435_fbd889ad539c.slice/crio-710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d WatchSource:0}: Error finding container 710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d: Status 404 returned error can't find the container with id 710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d Sep 10 21:24:20.117353 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:20.117322 2585 generic.go:356] "Generic (PLEG): container finished" podID="467dbc45-e1ae-4e04-9435-fbd889ad539c" containerID="24a1a180f499caad957cfba8e7cce171654a3198c195ded71bccff636fa8cc00" exitCode=0 Sep 10 21:24:20.117814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:20.117376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" event={"ID":"467dbc45-e1ae-4e04-9435-fbd889ad539c","Type":"ContainerDied","Data":"24a1a180f499caad957cfba8e7cce171654a3198c195ded71bccff636fa8cc00"} Sep 10 21:24:20.117814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:20.117428 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" event={"ID":"467dbc45-e1ae-4e04-9435-fbd889ad539c","Type":"ContainerStarted","Data":"710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d"} Sep 10 21:24:21.268029 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:21.267999 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:21.311077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:21.311045 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca\") pod \"467dbc45-e1ae-4e04-9435-fbd889ad539c\" (UID: \"467dbc45-e1ae-4e04-9435-fbd889ad539c\") " Sep 10 21:24:21.311362 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:21.311340 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "467dbc45-e1ae-4e04-9435-fbd889ad539c" (UID: "467dbc45-e1ae-4e04-9435-fbd889ad539c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:24:21.412711 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:21.412637 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467dbc45-e1ae-4e04-9435-fbd889ad539c-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:22.129546 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:22.129502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" event={"ID":"467dbc45-e1ae-4e04-9435-fbd889ad539c","Type":"ContainerDied","Data":"710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d"} Sep 10 21:24:22.129546 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:22.129520 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6" Sep 10 21:24:22.129546 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:22.129556 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710084eb5902e9c97ba11c20fc8815174f66c6b23a82b99651a74bce7ec20d8d" Sep 10 21:24:22.976772 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:22.976727 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 21:24:22.977255 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:22.976787 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 21:24:24.903422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:24.903388 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6"] Sep 10 21:24:24.906582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:24.906559 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6e4b80-9g5x6"] Sep 10 21:24:25.147718 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.147688 2585 generic.go:356] "Generic (PLEG): container finished" podID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerID="0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1" exitCode=0 Sep 10 21:24:25.147900 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.147759 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerDied","Data":"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1"} Sep 10 21:24:25.148207 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.148181 2585 scope.go:117] "RemoveContainer" containerID="0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1" Sep 10 21:24:25.149500 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.149474 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:24:25.149580 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.149563 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:24:25.149656 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:25.149641 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 21:24:26.155040 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:26.155006 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerStarted","Data":"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8"} Sep 10 21:24:26.667869 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:26.667831 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:26.668052 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:26.668033 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 21:24:26.668101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:26.668083 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 21:24:26.817912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:26.817877 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467dbc45-e1ae-4e04-9435-fbd889ad539c" path="/var/lib/kubelet/pods/467dbc45-e1ae-4e04-9435-fbd889ad539c/volumes" Sep 10 21:24:27.668553 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:27.668516 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 21:24:27.668958 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:27.668585 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 21:24:28.668101 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:28.668065 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 21:24:28.668281 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:28.668118 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 21:24:29.668328 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:29.668294 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 21:24:29.668719 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:29.668349 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 21:24:30.668444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:30.668406 2585 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 21:24:30.668860 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:30.668463 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 21:24:31.671041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:31.671012 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:31.671416 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:31.671377 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:31.672058 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:31.672034 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:31.787321 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:31.787289 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:24:32.191132 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:32.191098 2585 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" secret="" err="secret \"e2e-ipp-cleanup-6e4b80-openshift-default-dockercfg-xxjzm\" not found" Sep 10 21:24:33.195795 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:33.195750 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" containerID="cri-o://bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8" gracePeriod=30 Sep 10 21:24:35.290770 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.290731 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr"] Sep 10 21:24:35.291460 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.291440 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467dbc45-e1ae-4e04-9435-fbd889ad539c" containerName="revoke-keys" Sep 10 21:24:35.291508 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.291465 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="467dbc45-e1ae-4e04-9435-fbd889ad539c" containerName="revoke-keys" Sep 10 21:24:35.291583 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.291571 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="467dbc45-e1ae-4e04-9435-fbd889ad539c" containerName="revoke-keys" Sep 10 21:24:35.295273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.295254 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:35.298444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.298385 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:24:35.298444 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.298403 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:24:35.300668 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.300646 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr"] Sep 10 21:24:35.456523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.456489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr\" (UID: \"3efcac34-3e1b-4197-9814-f4b8eca1b9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:35.558008 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.557916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr\" (UID: \"3efcac34-3e1b-4197-9814-f4b8eca1b9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:35.558541 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.558520 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr\" (UID: \"3efcac34-3e1b-4197-9814-f4b8eca1b9a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:35.610324 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.610295 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:35.888605 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:35.888577 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr"] Sep 10 21:24:35.890403 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:24:35.890377 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efcac34_3e1b_4197_9814_f4b8eca1b9a8.slice/crio-8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c WatchSource:0}: Error finding container 8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c: Status 404 returned error can't find the container with id 8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c Sep 10 21:24:36.214097 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:36.214065 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" event={"ID":"3efcac34-3e1b-4197-9814-f4b8eca1b9a8","Type":"ContainerStarted","Data":"abd4d9f7eb6a50aa25d2e0495f46509d18743376511149e97b88b6ec58871e3a"} Sep 10 21:24:36.214253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:36.214120 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" event={"ID":"3efcac34-3e1b-4197-9814-f4b8eca1b9a8","Type":"ContainerStarted","Data":"8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c"} Sep 10 21:24:36.230653 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:36.230585 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" podStartSLOduration=1.23057066 podStartE2EDuration="1.23057066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:36.230096307 +0000 UTC m=+1559.938390692" watchObservedRunningTime="2026-09-10 21:24:36.23057066 +0000 UTC m=+1559.938865097" Sep 10 21:24:37.220606 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:37.220569 2585 generic.go:356] "Generic (PLEG): container finished" podID="3efcac34-3e1b-4197-9814-f4b8eca1b9a8" containerID="abd4d9f7eb6a50aa25d2e0495f46509d18743376511149e97b88b6ec58871e3a" exitCode=0 Sep 10 21:24:37.221029 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:37.220634 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" event={"ID":"3efcac34-3e1b-4197-9814-f4b8eca1b9a8","Type":"ContainerDied","Data":"abd4d9f7eb6a50aa25d2e0495f46509d18743376511149e97b88b6ec58871e3a"} Sep 10 21:24:38.363951 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.363926 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:38.456316 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.456262 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:38.488279 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.488255 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca\") pod \"3efcac34-3e1b-4197-9814-f4b8eca1b9a8\" (UID: \"3efcac34-3e1b-4197-9814-f4b8eca1b9a8\") " Sep 10 21:24:38.488659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.488627 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3efcac34-3e1b-4197-9814-f4b8eca1b9a8" (UID: "3efcac34-3e1b-4197-9814-f4b8eca1b9a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:24:38.589150 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589122 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589196 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589247 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589277 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589308 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589327 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589348 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvzp7\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589402 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589471 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert\") pod \"18141be7-30d4-47f1-a8ed-7f0e104316bb\" (UID: \"18141be7-30d4-47f1-a8ed-7f0e104316bb\") " Sep 10 21:24:38.589539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589486 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:24:38.589845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589728 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:24:38.589845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.589765 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:24:38.590074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590012 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.590074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590035 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.590074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590050 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3efcac34-3e1b-4197-9814-f4b8eca1b9a8-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.590074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590063 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.590074 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590030 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:24:38.590420 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.590080 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:24:38.591436 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.591406 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:24:38.591527 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.591480 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:24:38.591527 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.591487 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7" (OuterVolumeSpecName: "kube-api-access-wvzp7") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "kube-api-access-wvzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:24:38.591601 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.591526 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "18141be7-30d4-47f1-a8ed-7f0e104316bb" (UID: "18141be7-30d4-47f1-a8ed-7f0e104316bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:24:38.691471 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691429 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.691471 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691470 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.691471 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691480 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.691738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691491 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzp7\" (UniqueName: \"kubernetes.io/projected/18141be7-30d4-47f1-a8ed-7f0e104316bb-kube-api-access-wvzp7\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.691738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691500 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18141be7-30d4-47f1-a8ed-7f0e104316bb-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:38.691738 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:38.691508 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18141be7-30d4-47f1-a8ed-7f0e104316bb-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:24:39.232821 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.232791 2585 generic.go:356] "Generic (PLEG): container finished" podID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerID="bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8" exitCode=0 Sep 10 21:24:39.232979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.232835 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" Sep 10 21:24:39.232979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.232871 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerDied","Data":"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8"} Sep 10 21:24:39.232979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.232935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj" event={"ID":"18141be7-30d4-47f1-a8ed-7f0e104316bb","Type":"ContainerDied","Data":"86fdb303aeef08cd26dd809fa227ad9189e83f346ecf10a3e8d69643b1185bbb"} Sep 10 21:24:39.232979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.232959 2585 scope.go:117] "RemoveContainer" containerID="bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8" Sep 10 21:24:39.234338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.234219 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" event={"ID":"3efcac34-3e1b-4197-9814-f4b8eca1b9a8","Type":"ContainerDied","Data":"8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c"} Sep 10 21:24:39.234338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.234238 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr" Sep 10 21:24:39.234338 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.234250 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8898e053efc54afec46f05889958ebcbd1cbb8167282e39e408aa99b0a0bb85c" Sep 10 21:24:39.242357 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.242341 2585 scope.go:117] "RemoveContainer" containerID="0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1" Sep 10 21:24:39.250938 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.250682 2585 scope.go:117] "RemoveContainer" containerID="bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8" Sep 10 21:24:39.251179 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:24:39.251148 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8\": container with ID starting with bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8 not found: ID does not exist" containerID="bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8" Sep 10 21:24:39.251289 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.251188 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8"} err="failed to get container status \"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8\": rpc error: code = NotFound desc = could not find container \"bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8\": container with ID starting with bb2eae960e908ef0bf89c222ec3e8ebcf397cca7527c60409c19ec5392c0b6c8 not found: ID does not exist" Sep 10 21:24:39.251289 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.251211 2585 scope.go:117] "RemoveContainer" containerID="0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1" Sep 10 21:24:39.251488 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:24:39.251465 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1\": container with ID starting with 0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1 not found: ID does not exist" containerID="0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1" Sep 10 21:24:39.251577 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.251488 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1"} err="failed to get container status \"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1\": rpc error: code = NotFound desc = could not find container \"0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1\": container with ID starting with 0abb3e60e13a4ee9221f6d87eb2be4a56c9d944f55a617329de56344ba48e6a1 not found: ID does not exist" Sep 10 21:24:39.252604 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.252586 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:24:39.256442 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:39.256420 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6e4b80-openshift-default-5b46469d86-m68kj"] Sep 10 21:24:40.818998 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:40.818965 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" path="/var/lib/kubelet/pods/18141be7-30d4-47f1-a8ed-7f0e104316bb/volumes" Sep 10 21:24:51.241877 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:51.241843 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr"] Sep 10 21:24:51.257901 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:51.257875 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-84b69f-rwffr"] Sep 10 21:24:52.817797 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:24:52.817766 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3efcac34-3e1b-4197-9814-f4b8eca1b9a8" path="/var/lib/kubelet/pods/3efcac34-3e1b-4197-9814-f4b8eca1b9a8/volumes" Sep 10 21:25:41.323644 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.323599 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:25:41.324054 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.323829 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" containerID="cri-o://1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15" gracePeriod=30 Sep 10 21:25:41.690837 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.690758 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm"] Sep 10 21:25:41.691161 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691149 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:25:41.691209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691163 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:25:41.691209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691177 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efcac34-3e1b-4197-9814-f4b8eca1b9a8" containerName="revoke-keys" Sep 10 21:25:41.691209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691183 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efcac34-3e1b-4197-9814-f4b8eca1b9a8" containerName="revoke-keys" Sep 10 21:25:41.691303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691261 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:25:41.691303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691270 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efcac34-3e1b-4197-9814-f4b8eca1b9a8" containerName="revoke-keys" Sep 10 21:25:41.691303 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.691277 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:25:41.694256 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.694240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:41.699743 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.699720 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:25:41.699850 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.699774 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:25:41.709387 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.709366 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm"] Sep 10 21:25:41.744376 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.744355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm\" (UID: \"8ce0e845-1987-462a-9601-e2858f04c7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:41.845324 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.845300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm\" (UID: \"8ce0e845-1987-462a-9601-e2858f04c7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:41.845936 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:41.845913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm\" (UID: \"8ce0e845-1987-462a-9601-e2858f04c7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:42.004602 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:42.004527 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:42.278731 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:42.278706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm"] Sep 10 21:25:42.280917 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:25:42.280891 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce0e845_1987_462a_9601_e2858f04c7b5.slice/crio-6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab WatchSource:0}: Error finding container 6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab: Status 404 returned error can't find the container with id 6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab Sep 10 21:25:42.576919 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:42.576892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" event={"ID":"8ce0e845-1987-462a-9601-e2858f04c7b5","Type":"ContainerStarted","Data":"6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab"} Sep 10 21:25:43.584202 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:43.584167 2585 generic.go:356] "Generic (PLEG): container finished" podID="8ce0e845-1987-462a-9601-e2858f04c7b5" containerID="e4989540425154db1618d0644bc5d2383c61e9f4f0ba6e8ff0a5db1e6664681b" exitCode=0 Sep 10 21:25:43.584587 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:43.584242 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" event={"ID":"8ce0e845-1987-462a-9601-e2858f04c7b5","Type":"ContainerDied","Data":"e4989540425154db1618d0644bc5d2383c61e9f4f0ba6e8ff0a5db1e6664681b"} Sep 10 21:25:44.720857 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:44.720830 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:44.772201 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:44.772173 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca\") pod \"8ce0e845-1987-462a-9601-e2858f04c7b5\" (UID: \"8ce0e845-1987-462a-9601-e2858f04c7b5\") " Sep 10 21:25:44.772493 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:44.772471 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ce0e845-1987-462a-9601-e2858f04c7b5" (UID: "8ce0e845-1987-462a-9601-e2858f04c7b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:25:44.873815 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:44.873754 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce0e845-1987-462a-9601-e2858f04c7b5-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:45.596285 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:45.596252 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" Sep 10 21:25:45.596462 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:45.596283 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm" event={"ID":"8ce0e845-1987-462a-9601-e2858f04c7b5","Type":"ContainerDied","Data":"6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab"} Sep 10 21:25:45.596462 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:45.596329 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bdfccc700f0e406a2e8118c24f4a719646b27862d3da688f7fe5daa684bccab" Sep 10 21:25:46.584056 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.584025 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:25:46.602699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.602622 2585 generic.go:356] "Generic (PLEG): container finished" podID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerID="1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15" exitCode=0 Sep 10 21:25:46.602699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.602665 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" Sep 10 21:25:46.602699 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.602683 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerDied","Data":"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15"} Sep 10 21:25:46.602893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.602752 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb" event={"ID":"79ebc914-92d4-4fff-b945-6412b7ba457d","Type":"ContainerDied","Data":"3a71988a55a8d99be913a2285e54d5a8495cceb5d55bd13749a1bd3412dc2de6"} Sep 10 21:25:46.602893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.602776 2585 scope.go:117] "RemoveContainer" containerID="1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15" Sep 10 21:25:46.613659 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.613642 2585 scope.go:117] "RemoveContainer" containerID="767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58" Sep 10 21:25:46.622088 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.622065 2585 scope.go:117] "RemoveContainer" containerID="1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15" Sep 10 21:25:46.622406 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:25:46.622384 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15\": container with ID starting with 1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15 not found: ID does not exist" containerID="1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15" Sep 10 21:25:46.622505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.622414 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15"} err="failed to get container status \"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15\": rpc error: code = NotFound desc = could not find container \"1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15\": container with ID starting with 1bc3ba14e6ed1d5c2e69d2035f84f2b9b878ed4701b6a0b24f965afb399bda15 not found: ID does not exist" Sep 10 21:25:46.622505 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.622432 2585 scope.go:117] "RemoveContainer" containerID="767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58" Sep 10 21:25:46.622748 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:25:46.622730 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58\": container with ID starting with 767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58 not found: ID does not exist" containerID="767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58" Sep 10 21:25:46.622806 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.622755 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58"} err="failed to get container status \"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58\": rpc error: code = NotFound desc = could not find container \"767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58\": container with ID starting with 767078e73ec11c8c22b0e67a83463fcb9f3648a296b5f6eda04f7295643a8a58 not found: ID does not exist" Sep 10 21:25:46.690690 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690663 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.690801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690717 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.690801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690739 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.690801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690763 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.690801 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690785 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.691027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690808 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.691027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690835 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldcdt\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.691027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690878 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.691027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.690926 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket\") pod \"79ebc914-92d4-4fff-b945-6412b7ba457d\" (UID: \"79ebc914-92d4-4fff-b945-6412b7ba457d\") " Sep 10 21:25:46.691184 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691100 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data" (OuterVolumeSpecName: "istio-data") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:25:46.691239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691210 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:25:46.691239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691217 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:25:46.691345 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691268 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.691345 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691299 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:25:46.691450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.691344 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:25:46.692789 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.692766 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:25:46.693080 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.693066 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:25:46.693137 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.693100 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token" (OuterVolumeSpecName: "istio-token") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:25:46.693253 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.693235 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt" (OuterVolumeSpecName: "kube-api-access-ldcdt") pod "79ebc914-92d4-4fff-b945-6412b7ba457d" (UID: "79ebc914-92d4-4fff-b945-6412b7ba457d"). InnerVolumeSpecName "kube-api-access-ldcdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:25:46.792098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792072 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldcdt\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-kube-api-access-ldcdt\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792094 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792098 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792104 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792113 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792121 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ebc914-92d4-4fff-b945-6412b7ba457d-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792129 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792136 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ebc914-92d4-4fff-b945-6412b7ba457d-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.792282 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.792144 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ebc914-92d4-4fff-b945-6412b7ba457d-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:25:46.919606 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.919540 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:25:46.922197 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:46.922177 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-84b69f-openshift-default-6dc4cd79d4-v6ksb"] Sep 10 21:25:48.818045 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:48.818014 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" path="/var/lib/kubelet/pods/79ebc914-92d4-4fff-b945-6412b7ba457d/volumes" Sep 10 21:25:52.433412 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:52.433369 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm"] Sep 10 21:25:52.435336 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:52.435309 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-868994-s8kzm"] Sep 10 21:25:52.817195 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:52.817167 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce0e845-1987-462a-9601-e2858f04c7b5" path="/var/lib/kubelet/pods/8ce0e845-1987-462a-9601-e2858f04c7b5/volumes" Sep 10 21:25:59.710355 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:59.710324 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:25:59.710772 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:25:59.710555 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" containerID="cri-o://388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa" gracePeriod=30 Sep 10 21:26:00.097450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097412 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh"] Sep 10 21:26:00.097829 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097816 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097830 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18141be7-30d4-47f1-a8ed-7f0e104316bb" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097855 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097861 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097868 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097874 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097880 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce0e845-1987-462a-9601-e2858f04c7b5" containerName="revoke-keys" Sep 10 21:26:00.097929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097885 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce0e845-1987-462a-9601-e2858f04c7b5" containerName="revoke-keys" Sep 10 21:26:00.098141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097938 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:00.098141 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.097945 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce0e845-1987-462a-9601-e2858f04c7b5" containerName="revoke-keys" Sep 10 21:26:00.102209 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.102193 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:00.104986 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.104964 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:26:00.105103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.104966 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:26:00.111222 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.111196 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh"] Sep 10 21:26:00.206207 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.206180 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh\" (UID: \"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:00.306889 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.306861 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh\" (UID: \"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:00.307445 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.307426 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh\" (UID: \"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:00.414212 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.414126 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:00.691826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:00.691800 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh"] Sep 10 21:26:00.693546 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:26:00.693516 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66a2c3f_f9ec_4ca0_bf0b_ff6c858ace7f.slice/crio-e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3 WatchSource:0}: Error finding container e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3: Status 404 returned error can't find the container with id e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3 Sep 10 21:26:01.686732 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:01.686642 2585 generic.go:356] "Generic (PLEG): container finished" podID="b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" containerID="2a7d262ac4121d8896830066fba599b3ceecdad1b531c11e46cfb0a0bda02c5b" exitCode=0 Sep 10 21:26:01.687082 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:01.686737 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" event={"ID":"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f","Type":"ContainerDied","Data":"2a7d262ac4121d8896830066fba599b3ceecdad1b531c11e46cfb0a0bda02c5b"} Sep 10 21:26:01.687082 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:01.686791 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" event={"ID":"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f","Type":"ContainerStarted","Data":"e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3"} Sep 10 21:26:02.838023 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:02.837999 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:02.934102 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:02.934076 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca\") pod \"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f\" (UID: \"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f\") " Sep 10 21:26:02.934407 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:02.934383 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" (UID: "b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:26:03.035516 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.035478 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:03.526514 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.526474 2585 patch_prober.go:28] interesting pod/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 21:26:03.526692 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.526558 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:26:03.699062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.699031 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" Sep 10 21:26:03.699062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.699056 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh" event={"ID":"b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f","Type":"ContainerDied","Data":"e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3"} Sep 10 21:26:03.699273 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:03.699102 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8d8e0ecaf93f71d67418e5e2c22a57b77deb3375f9ef4cef48479a368d614e3" Sep 10 21:26:04.952962 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:04.952940 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:26:05.053046 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053021 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053073 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053103 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfwb\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053142 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053190 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053183 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053392 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053215 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053392 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053280 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053392 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053375 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:05.053544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053418 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053447 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token\") pod \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\" (UID: \"011ee9b7-4977-4bc0-8b39-6fee4e939ed4\") " Sep 10 21:26:05.053544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053444 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:05.053544 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053502 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data" (OuterVolumeSpecName: "istio-data") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:05.053790 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053632 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:26:05.053790 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053718 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:05.053979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053946 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.054041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053984 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.054041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.053999 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.054041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.054013 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.054041 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.054026 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.055416 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.055390 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token" (OuterVolumeSpecName: "istio-token") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:26:05.055523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.055405 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:26:05.055523 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.055424 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:05.055597 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.055527 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb" (OuterVolumeSpecName: "kube-api-access-rqfwb") pod "011ee9b7-4977-4bc0-8b39-6fee4e939ed4" (UID: "011ee9b7-4977-4bc0-8b39-6fee4e939ed4"). InnerVolumeSpecName "kube-api-access-rqfwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:26:05.154557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.154529 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.154557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.154552 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.154557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.154562 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.154750 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.154570 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfwb\" (UniqueName: \"kubernetes.io/projected/011ee9b7-4977-4bc0-8b39-6fee4e939ed4-kube-api-access-rqfwb\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:05.714942 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.714911 2585 generic.go:356] "Generic (PLEG): container finished" podID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerID="388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa" exitCode=0 Sep 10 21:26:05.715085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.714963 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" Sep 10 21:26:05.715085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.714986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerDied","Data":"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa"} Sep 10 21:26:05.715085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.715028 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r" event={"ID":"011ee9b7-4977-4bc0-8b39-6fee4e939ed4","Type":"ContainerDied","Data":"ea013a72d99089285df44f6af5f7a121c8d785a9d04c57ac1c7dd3c99f7b215f"} Sep 10 21:26:05.715085 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.715050 2585 scope.go:117] "RemoveContainer" containerID="388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa" Sep 10 21:26:05.724378 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.724361 2585 scope.go:117] "RemoveContainer" containerID="cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0" Sep 10 21:26:05.732302 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.732285 2585 scope.go:117] "RemoveContainer" containerID="388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa" Sep 10 21:26:05.732549 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:26:05.732528 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa\": container with ID starting with 388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa not found: ID does not exist" containerID="388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa" Sep 10 21:26:05.732599 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.732557 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa"} err="failed to get container status \"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa\": rpc error: code = NotFound desc = could not find container \"388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa\": container with ID starting with 388f77d25e40d8f081c8efac86b85218cbfa35548e731f16e5f4c3a1bc317efa not found: ID does not exist" Sep 10 21:26:05.732599 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.732574 2585 scope.go:117] "RemoveContainer" containerID="cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0" Sep 10 21:26:05.732842 ip-10-0-130-192 kubenswrapper[2585]: E0910 21:26:05.732826 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0\": container with ID starting with cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0 not found: ID does not exist" containerID="cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0" Sep 10 21:26:05.732882 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.732849 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0"} err="failed to get container status \"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0\": rpc error: code = NotFound desc = could not find container \"cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0\": container with ID starting with cffa8c69829dddc4185247fa89d21a50e25c78b6ece7dab5d9e93fceaea927e0 not found: ID does not exist" Sep 10 21:26:05.739482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.739460 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:26:05.743571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:05.743550 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-868994-openshift-default-599dd7cf9b-mtq7r"] Sep 10 21:26:06.817578 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:06.817538 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" path="/var/lib/kubelet/pods/011ee9b7-4977-4bc0-8b39-6fee4e939ed4/volumes" Sep 10 21:26:11.291330 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:11.291294 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh"] Sep 10 21:26:11.293567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:11.293545 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-14acb0-vz9qh"] Sep 10 21:26:12.817557 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:12.817523 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" path="/var/lib/kubelet/pods/b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f/volumes" Sep 10 21:26:18.482881 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.482848 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh"] Sep 10 21:26:18.483245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483206 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:26:18.483245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483217 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:26:18.483245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483238 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" containerName="revoke-keys" Sep 10 21:26:18.483245 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483243 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" containerName="revoke-keys" Sep 10 21:26:18.483381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483311 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66a2c3f-f9ec-4ca0-bf0b-ff6c858ace7f" containerName="revoke-keys" Sep 10 21:26:18.483381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483319 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:26:18.483381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483328 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:26:18.483381 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.483337 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ebc914-92d4-4fff-b945-6412b7ba457d" containerName="istio-proxy" Sep 10 21:26:18.486586 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.486569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:18.498072 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.498045 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:26:18.498285 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.498270 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:26:18.505735 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.505711 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh"] Sep 10 21:26:18.573024 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.572994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh\" (UID: \"83276a01-9bfd-4ec2-a3ed-72579ac25751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:18.673677 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.673635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh\" (UID: \"83276a01-9bfd-4ec2-a3ed-72579ac25751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:18.674315 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.674287 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh\" (UID: \"83276a01-9bfd-4ec2-a3ed-72579ac25751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:18.796087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:18.796053 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:19.072347 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:19.072321 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh"] Sep 10 21:26:19.074562 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:26:19.074534 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83276a01_9bfd_4ec2_a3ed_72579ac25751.slice/crio-cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf WatchSource:0}: Error finding container cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf: Status 404 returned error can't find the container with id cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf Sep 10 21:26:19.790287 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:19.790262 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" event={"ID":"83276a01-9bfd-4ec2-a3ed-72579ac25751","Type":"ContainerStarted","Data":"15638c1e87eb0ab8bacdc0be89022dd5513256a24984e35de56bec79c79c6830"} Sep 10 21:26:19.790571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:19.790310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" event={"ID":"83276a01-9bfd-4ec2-a3ed-72579ac25751","Type":"ContainerStarted","Data":"cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf"} Sep 10 21:26:20.796633 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:20.796571 2585 generic.go:356] "Generic (PLEG): container finished" podID="83276a01-9bfd-4ec2-a3ed-72579ac25751" containerID="15638c1e87eb0ab8bacdc0be89022dd5513256a24984e35de56bec79c79c6830" exitCode=0 Sep 10 21:26:20.797018 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:20.796659 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" event={"ID":"83276a01-9bfd-4ec2-a3ed-72579ac25751","Type":"ContainerDied","Data":"15638c1e87eb0ab8bacdc0be89022dd5513256a24984e35de56bec79c79c6830"} Sep 10 21:26:20.923746 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:20.923719 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:20.996567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:20.996533 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca\") pod \"83276a01-9bfd-4ec2-a3ed-72579ac25751\" (UID: \"83276a01-9bfd-4ec2-a3ed-72579ac25751\") " Sep 10 21:26:20.996893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:20.996871 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83276a01-9bfd-4ec2-a3ed-72579ac25751" (UID: "83276a01-9bfd-4ec2-a3ed-72579ac25751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:26:21.098027 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:21.097951 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83276a01-9bfd-4ec2-a3ed-72579ac25751-maas-api-service-ca\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:21.803219 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:21.803184 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" Sep 10 21:26:21.803219 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:21.803194 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh" event={"ID":"83276a01-9bfd-4ec2-a3ed-72579ac25751","Type":"ContainerDied","Data":"cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf"} Sep 10 21:26:21.803685 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:21.803263 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cefa29fa80f8d2e5e628b81bb9d0ed0b4221dd2e321068332b8bc343fb8667cf" Sep 10 21:26:29.303105 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:29.303073 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh"] Sep 10 21:26:29.306293 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:29.306267 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-755926-ngswh"] Sep 10 21:26:30.818033 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:30.817999 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83276a01-9bfd-4ec2-a3ed-72579ac25751" path="/var/lib/kubelet/pods/83276a01-9bfd-4ec2-a3ed-72579ac25751/volumes" Sep 10 21:26:36.784811 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:36.784779 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:26:36.785239 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:36.784967 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" containerID="cri-o://831d5ce91458b24844beb2afb9fbc6125af2c8ff50a236ba07a8429b10f2c19c" gracePeriod=30 Sep 10 21:26:41.911114 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:41.911073 2585 generic.go:356] "Generic (PLEG): container finished" podID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerID="831d5ce91458b24844beb2afb9fbc6125af2c8ff50a236ba07a8429b10f2c19c" exitCode=0 Sep 10 21:26:41.911567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:41.911155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerDied","Data":"831d5ce91458b24844beb2afb9fbc6125af2c8ff50a236ba07a8429b10f2c19c"} Sep 10 21:26:41.911567 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:41.911222 2585 scope.go:117] "RemoveContainer" containerID="5483838bf85fb2dab16149ce8280208f9940caa5d7303e07af698ad527f28726" Sep 10 21:26:42.053391 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.053328 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:26:42.195769 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195738 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.195963 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195777 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.195963 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195807 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.195963 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195841 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.195963 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195875 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.196181 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.195980 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.196181 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196043 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxnsj\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.196181 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196093 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:42.196181 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196075 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data" (OuterVolumeSpecName: "istio-data") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:42.196181 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196120 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:42.196441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196217 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.196441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196245 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token\") pod \"e018a895-59a0-4c6d-b9f7-0df36808013d\" (UID: \"e018a895-59a0-4c6d-b9f7-0df36808013d\") " Sep 10 21:26:42.196441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196252 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:42.196441 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196354 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:26:42.196658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196584 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-credential-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.196658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196605 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-data\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.196658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196640 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-socket\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.196658 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196654 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e018a895-59a0-4c6d-b9f7-0df36808013d-istiod-ca-cert\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.196828 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.196667 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-workload-certs\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.198823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.198790 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:26:42.198823 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.198814 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:26:42.198967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.198821 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj" (OuterVolumeSpecName: "kube-api-access-cxnsj") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "kube-api-access-cxnsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:26:42.198967 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.198874 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token" (OuterVolumeSpecName: "istio-token") pod "e018a895-59a0-4c6d-b9f7-0df36808013d" (UID: "e018a895-59a0-4c6d-b9f7-0df36808013d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:26:42.297657 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.297596 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnsj\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-kube-api-access-cxnsj\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.297657 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.297653 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-envoy\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.297893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.297669 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-token\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.297893 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.297683 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e018a895-59a0-4c6d-b9f7-0df36808013d-istio-podinfo\") on node \"ip-10-0-130-192.ec2.internal\" DevicePath \"\"" Sep 10 21:26:42.917872 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.917839 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" event={"ID":"e018a895-59a0-4c6d-b9f7-0df36808013d","Type":"ContainerDied","Data":"d92dd5abba9ab206609dd1863f9cfc46c230f0fd790e37d9223f89b063a2a129"} Sep 10 21:26:42.918249 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.917899 2585 scope.go:117] "RemoveContainer" containerID="831d5ce91458b24844beb2afb9fbc6125af2c8ff50a236ba07a8429b10f2c19c" Sep 10 21:26:42.918249 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.917847 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt" Sep 10 21:26:42.940076 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.940051 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:26:42.944713 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:42.944691 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-755926-openshift-default-69cf84f7c8-8vhnt"] Sep 10 21:26:44.817674 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:26:44.817643 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" path="/var/lib/kubelet/pods/e018a895-59a0-4c6d-b9f7-0df36808013d/volumes" Sep 10 21:27:37.131077 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:27:37.131045 2585 scope.go:117] "RemoveContainer" containerID="ec8c914c66be9e0276ed87bb215148085f261a9c1a931842072de11ac7738e03" Sep 10 21:27:37.139678 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:27:37.139658 2585 scope.go:117] "RemoveContainer" containerID="30d3410cd96b1a4bd132d350944c750ace8d0a9da412956bdf71baa42487eedb" Sep 10 21:27:37.147661 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:27:37.147640 2585 scope.go:117] "RemoveContainer" containerID="f5a4928f5715816bc11a51af8a2315620fe8c079826c15b3c0aa05b46e3ebe1b" Sep 10 21:28:36.919045 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:28:36.919014 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:28:36.920777 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:28:36.920757 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:28:36.926585 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:28:36.926563 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:28:36.928666 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:28:36.928647 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:28:37.177929 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:28:37.177845 2585 scope.go:117] "RemoveContainer" containerID="c363bc12d57d6689bcbae47e466e0a8edde155215bc84e8db7c3c03fe0711dbb" Sep 10 21:30:00.151422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151338 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2"] Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151712 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151725 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151738 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83276a01-9bfd-4ec2-a3ed-72579ac25751" containerName="revoke-keys" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151744 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="83276a01-9bfd-4ec2-a3ed-72579ac25751" containerName="revoke-keys" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151757 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151763 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="011ee9b7-4977-4bc0-8b39-6fee4e939ed4" containerName="istio-proxy" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151818 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:30:00.153813 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.151827 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="83276a01-9bfd-4ec2-a3ed-72579ac25751" containerName="revoke-keys" Sep 10 21:30:00.154816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.154800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" Sep 10 21:30:00.159001 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.158982 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8q9zm\"" Sep 10 21:30:00.163837 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.163818 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" Sep 10 21:30:00.167005 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.166706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2"] Sep 10 21:30:00.436222 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.436197 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2"] Sep 10 21:30:00.438376 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:30:00.438350 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f512068_003c_43fb_85b4_2944468c68d0.slice/crio-f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810 WatchSource:0}: Error finding container f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810: Status 404 returned error can't find the container with id f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810 Sep 10 21:30:00.916819 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.916786 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" event={"ID":"6f512068-003c-43fb-85b4-2944468c68d0","Type":"ContainerStarted","Data":"5a3241ac010403dc7bfa82dd0dc101a7bc805d612e1fd1bfb9890d4cdded0adf"} Sep 10 21:30:00.916959 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.916830 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" event={"ID":"6f512068-003c-43fb-85b4-2944468c68d0","Type":"ContainerStarted","Data":"f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810"} Sep 10 21:30:00.933330 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:00.933281 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" podStartSLOduration=0.933267039 podStartE2EDuration="933.267039ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:30:00.931215681 +0000 UTC m=+1884.639510079" watchObservedRunningTime="2026-09-10 21:30:00.933267039 +0000 UTC m=+1884.641561426" Sep 10 21:30:01.923183 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:01.923148 2585 generic.go:356] "Generic (PLEG): container finished" podID="6f512068-003c-43fb-85b4-2944468c68d0" containerID="5a3241ac010403dc7bfa82dd0dc101a7bc805d612e1fd1bfb9890d4cdded0adf" exitCode=0 Sep 10 21:30:01.923564 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:01.923190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" event={"ID":"6f512068-003c-43fb-85b4-2944468c68d0","Type":"ContainerDied","Data":"5a3241ac010403dc7bfa82dd0dc101a7bc805d612e1fd1bfb9890d4cdded0adf"} Sep 10 21:30:03.056091 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:03.056067 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" Sep 10 21:30:03.934089 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:03.934059 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" Sep 10 21:30:03.934259 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:03.934095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-2wtg2" event={"ID":"6f512068-003c-43fb-85b4-2944468c68d0","Type":"ContainerDied","Data":"f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810"} Sep 10 21:30:03.934259 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:03.934140 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90d3d56597fca315b9f53f6a6065290851f60038aec238529daf75c8b44a810" Sep 10 21:30:37.203558 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:37.203524 2585 scope.go:117] "RemoveContainer" containerID="24a1a180f499caad957cfba8e7cce171654a3198c195ded71bccff636fa8cc00" Sep 10 21:30:37.211814 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:30:37.211788 2585 scope.go:117] "RemoveContainer" containerID="abd4d9f7eb6a50aa25d2e0495f46509d18743376511149e97b88b6ec58871e3a" Sep 10 21:32:37.240987 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:32:37.240955 2585 scope.go:117] "RemoveContainer" containerID="2a7d262ac4121d8896830066fba599b3ceecdad1b531c11e46cfb0a0bda02c5b" Sep 10 21:32:37.251467 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:32:37.251426 2585 scope.go:117] "RemoveContainer" containerID="e4989540425154db1618d0644bc5d2383c61e9f4f0ba6e8ff0a5db1e6664681b" Sep 10 21:32:37.259010 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:32:37.258992 2585 scope.go:117] "RemoveContainer" containerID="15638c1e87eb0ab8bacdc0be89022dd5513256a24984e35de56bec79c79c6830" Sep 10 21:33:08.455119 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:08.455034 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zwfzw_347b189d-6114-4d35-ba84-d44240c7e302/server/0.log" Sep 10 21:33:08.565103 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:08.565073 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7bl8x_2c469da6-d7ca-448a-af8c-d1409eb1203d/manager/0.log" Sep 10 21:33:08.806595 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:08.806567 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d69f6bb5-lpv78_3ccc173c-d65c-4efc-8736-37bc6ebf0855/copy-manifests/0.log" Sep 10 21:33:08.813728 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:08.813705 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d69f6bb5-lpv78_3ccc173c-d65c-4efc-8736-37bc6ebf0855/manager/0.log" Sep 10 21:33:09.481851 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:09.481824 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817915-trb96_bc197ab4-5383-49e3-b660-a44f7fde9689/cleanup/0.log" Sep 10 21:33:09.582912 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:09.582885 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817930-2wtg2_6f512068-003c-43fb-85b4-2944468c68d0/cleanup/0.log" Sep 10 21:33:11.506623 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:11.506590 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws_9541fd13-ec64-49cf-85bc-638adf10a876/istio-proxy/0.log" Sep 10 21:33:11.618206 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:11.618174 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cwlcv_a80e7f97-2f59-4963-8fe8-bac43d6b0f7e/discovery/0.log" Sep 10 21:33:11.954769 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:11.954741 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/1.log" Sep 10 21:33:12.750180 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:12.750152 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wkdzt_52bc01d2-7ba9-4029-9915-bffdcdbb351c/main/0.log" Sep 10 21:33:12.968419 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:12.968393 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gwf2z_bedcec28-0fa8-464c-8236-d8837a971d91/main/0.log" Sep 10 21:33:13.191950 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:13.191922 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mxnfv_09633c8d-94cc-499d-88e6-bae86a394555/main/0.log" Sep 10 21:33:21.373070 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:21.373038 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-trv4s_0f6cb7b1-8cc9-438e-b98f-d5b70e2935f0/global-pull-secret-syncer/0.log" Sep 10 21:33:21.478073 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:21.478042 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dj2ws_9439980b-2015-4a42-b147-01b0711cf26f/konnectivity-agent/0.log" Sep 10 21:33:21.583571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:21.583543 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-192.ec2.internal_b203a93e7590a5afc1a736ff688dacdd/haproxy/0.log" Sep 10 21:33:28.270309 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.270271 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/alertmanager/0.log" Sep 10 21:33:28.307236 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.307209 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/config-reloader/0.log" Sep 10 21:33:28.334363 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.334340 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/kube-rbac-proxy-web/0.log" Sep 10 21:33:28.365411 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.365392 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/kube-rbac-proxy/0.log" Sep 10 21:33:28.389763 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.389743 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/kube-rbac-proxy-metric/0.log" Sep 10 21:33:28.412972 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.412952 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/prom-label-proxy/0.log" Sep 10 21:33:28.436163 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.436142 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c72d390b-040e-48ef-94dd-3840575c250b/init-config-reloader/0.log" Sep 10 21:33:28.677514 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.677484 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-2l2fr_8e85a6da-560f-4e4c-b760-adc779d7f37a/monitoring-plugin/0.log" Sep 10 21:33:28.836155 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.836075 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666sp_7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d/node-exporter/0.log" Sep 10 21:33:28.863767 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.863744 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666sp_7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d/kube-rbac-proxy/0.log" Sep 10 21:33:28.891498 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:28.891481 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666sp_7c0132b3-2f37-4f0b-9a87-b9b51bcbfa9d/init-textfile/0.log" Sep 10 21:33:29.476075 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.476048 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/thanos-query/0.log" Sep 10 21:33:29.502010 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.501986 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/kube-rbac-proxy-web/0.log" Sep 10 21:33:29.525054 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.525035 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/kube-rbac-proxy/0.log" Sep 10 21:33:29.548717 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.548690 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/prom-label-proxy/0.log" Sep 10 21:33:29.571172 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.571150 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/kube-rbac-proxy-rules/0.log" Sep 10 21:33:29.594200 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:29.594179 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5565f8586f-dgv59_42aa9918-064e-42c8-951f-7e3cfa20ad97/kube-rbac-proxy-metrics/0.log" Sep 10 21:33:30.884481 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:30.884452 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-vjq5j_9c8ec021-c528-4678-be74-66f8da1edd93/networking-console-plugin/0.log" Sep 10 21:33:31.375295 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375263 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z"] Sep 10 21:33:31.375647 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375631 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f512068-003c-43fb-85b4-2944468c68d0" containerName="cleanup" Sep 10 21:33:31.375702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375649 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f512068-003c-43fb-85b4-2944468c68d0" containerName="cleanup" Sep 10 21:33:31.375702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375681 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:33:31.375702 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375687 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:33:31.375805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375739 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f512068-003c-43fb-85b4-2944468c68d0" containerName="cleanup" Sep 10 21:33:31.375805 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.375751 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e018a895-59a0-4c6d-b9f7-0df36808013d" containerName="istio-proxy" Sep 10 21:33:31.378983 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.378961 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.381314 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.381293 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jkhl9\"/\"kube-root-ca.crt\"" Sep 10 21:33:31.381403 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.381295 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jkhl9\"/\"default-dockercfg-n4mbj\"" Sep 10 21:33:31.382050 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.382035 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jkhl9\"/\"openshift-service-ca.crt\"" Sep 10 21:33:31.388223 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.388202 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z"] Sep 10 21:33:31.456326 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.456302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-podres\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.456447 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.456336 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-sys\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.456447 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.456364 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-lib-modules\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.456562 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.456463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72ck\" (UniqueName: \"kubernetes.io/projected/e5fee058-9a09-4833-8008-87a2d459f6f3-kube-api-access-d72ck\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.456562 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.456499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-proc\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.538582 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.538557 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:33:31.542348 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.542322 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/2.log" Sep 10 21:33:31.557421 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-podres\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-sys\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557476 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-lib-modules\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557539 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-podres\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557546 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72ck\" (UniqueName: \"kubernetes.io/projected/e5fee058-9a09-4833-8008-87a2d459f6f3-kube-api-access-d72ck\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557578 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-sys\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-proc\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-lib-modules\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.557747 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.557695 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5fee058-9a09-4833-8008-87a2d459f6f3-proc\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.565482 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.565456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72ck\" (UniqueName: \"kubernetes.io/projected/e5fee058-9a09-4833-8008-87a2d459f6f3-kube-api-access-d72ck\") pod \"perf-node-gather-daemonset-hdl6z\" (UID: \"e5fee058-9a09-4833-8008-87a2d459f6f3\") " pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.689988 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.689912 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:31.956787 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:31.956759 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z"] Sep 10 21:33:31.958931 ip-10-0-130-192 kubenswrapper[2585]: W0910 21:33:31.958903 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode5fee058_9a09_4833_8008_87a2d459f6f3.slice/crio-ea3962fc5712185683f7acfabda654833674a2bbdcb7bd5ce32acb9165388ddd WatchSource:0}: Error finding container ea3962fc5712185683f7acfabda654833674a2bbdcb7bd5ce32acb9165388ddd: Status 404 returned error can't find the container with id ea3962fc5712185683f7acfabda654833674a2bbdcb7bd5ce32acb9165388ddd Sep 10 21:33:32.855571 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:32.855543 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-7zl77_7d63a201-9b23-4e47-b17f-e60883195730/volume-data-source-validator/0.log" Sep 10 21:33:32.959715 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:32.959675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" event={"ID":"e5fee058-9a09-4833-8008-87a2d459f6f3","Type":"ContainerStarted","Data":"f488bc39459a1b5101e4dd61c9282dc3b82cd18220a9d5b58dd7daf791d300de"} Sep 10 21:33:32.960087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:32.959777 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:32.960087 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:32.959807 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" event={"ID":"e5fee058-9a09-4833-8008-87a2d459f6f3","Type":"ContainerStarted","Data":"ea3962fc5712185683f7acfabda654833674a2bbdcb7bd5ce32acb9165388ddd"} Sep 10 21:33:32.977417 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:32.977371 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" podStartSLOduration=1.9773571410000002 podStartE2EDuration="1.977357141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:32.975653902 +0000 UTC m=+2096.683948287" watchObservedRunningTime="2026-09-10 21:33:32.977357141 +0000 UTC m=+2096.685651526" Sep 10 21:33:34.031783 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:34.031758 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t6h5z_6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9/dns/0.log" Sep 10 21:33:34.054013 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:34.053993 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t6h5z_6a2cf1ee-cd72-44b5-b63a-3fb8ef00aae9/kube-rbac-proxy/0.log" Sep 10 21:33:34.105344 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:34.105321 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5hf9_60a40174-e77b-4d74-b7d2-54a73dc950ae/dns-node-resolver/0.log" Sep 10 21:33:34.700562 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:34.700533 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gvzb_b5526c20-190b-471e-9912-4e50d4f40937/node-ca/0.log" Sep 10 21:33:35.727918 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:35.727892 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws_9541fd13-ec64-49cf-85bc-638adf10a876/istio-proxy/0.log" Sep 10 21:33:35.943787 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:35.943752 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cwlcv_a80e7f97-2f59-4963-8fe8-bac43d6b0f7e/discovery/0.log" Sep 10 21:33:36.028559 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.028529 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/0.log" Sep 10 21:33:36.037096 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.037073 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/1.log" Sep 10 21:33:36.955817 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.955787 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:33:36.959242 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.959219 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-pf8tm_2555a5e5-54fd-48e0-907a-4aef7afcb35c/console-operator/1.log" Sep 10 21:33:36.963531 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.963512 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:33:36.967129 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:36.967114 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g68vl_ae84f37d-5926-44bd-9815-a986b2fc28c0/ovn-acl-logging/0.log" Sep 10 21:33:37.076790 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.076766 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zxhqt_33e80727-2e90-4091-84aa-90798cffb324/serve-healthcheck-canary/0.log" Sep 10 21:33:37.565538 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.565501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-x964d_4c728a21-5a9c-49d4-8e05-dbdbc6ded43a/insights-operator/0.log" Sep 10 21:33:37.567062 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.567042 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-x964d_4c728a21-5a9c-49d4-8e05-dbdbc6ded43a/insights-operator/1.log" Sep 10 21:33:37.732642 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.732603 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbf6p_6abe2e34-0083-40d6-bd26-12d10effbbe0/kube-rbac-proxy/0.log" Sep 10 21:33:37.754845 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.754822 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbf6p_6abe2e34-0083-40d6-bd26-12d10effbbe0/exporter/0.log" Sep 10 21:33:37.779667 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:37.779637 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbf6p_6abe2e34-0083-40d6-bd26-12d10effbbe0/extractor/0.log" Sep 10 21:33:38.973629 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:38.973583 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jkhl9/perf-node-gather-daemonset-hdl6z" Sep 10 21:33:40.456241 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:40.456216 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zwfzw_347b189d-6114-4d35-ba84-d44240c7e302/server/0.log" Sep 10 21:33:40.538004 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:40.537982 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7bl8x_2c469da6-d7ca-448a-af8c-d1409eb1203d/manager/0.log" Sep 10 21:33:40.738532 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:40.738451 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d69f6bb5-lpv78_3ccc173c-d65c-4efc-8736-37bc6ebf0855/manager/0.log" Sep 10 21:33:40.762405 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:40.762382 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d69f6bb5-lpv78_3ccc173c-d65c-4efc-8736-37bc6ebf0855/copy-manifests/0.log" Sep 10 21:33:42.151097 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:42.151062 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws_9541fd13-ec64-49cf-85bc-638adf10a876/istio-proxy/0.log" Sep 10 21:33:42.170450 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:42.170411 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79fcf6dff7-zqcs7_cbc0fc5d-1a1c-4ec3-ad2d-f03c95b70c2c/manager/0.log" Sep 10 21:33:42.360954 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:42.360926 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cwlcv_a80e7f97-2f59-4963-8fe8-bac43d6b0f7e/discovery/0.log" Sep 10 21:33:42.407979 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:42.407909 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/0.log" Sep 10 21:33:42.416192 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:42.416169 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/1.log" Sep 10 21:33:43.998966 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:43.998937 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kksws_9541fd13-ec64-49cf-85bc-638adf10a876/istio-proxy/0.log" Sep 10 21:33:44.219816 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:44.219785 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cwlcv_a80e7f97-2f59-4963-8fe8-bac43d6b0f7e/discovery/0.log" Sep 10 21:33:44.278188 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:44.278159 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/0.log" Sep 10 21:33:44.289170 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:44.289146 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-fg5pc_8e0894d6-a8cf-4228-98f4-6bdf22536405/istio-proxy/1.log" Sep 10 21:33:48.852748 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.852720 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2gmjj_3487b180-4035-4638-b5bd-97a5dcc6501f/kube-multus/0.log" Sep 10 21:33:48.876125 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.876101 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/kube-multus-additional-cni-plugins/0.log" Sep 10 21:33:48.902327 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.902308 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/egress-router-binary-copy/0.log" Sep 10 21:33:48.926422 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.926404 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/cni-plugins/0.log" Sep 10 21:33:48.950798 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.950780 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/bond-cni-plugin/0.log" Sep 10 21:33:48.975124 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.975103 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/routeoverride-cni/0.log" Sep 10 21:33:48.998826 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:48.998808 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/whereabouts-cni-bincopy/0.log" Sep 10 21:33:49.022943 ip-10-0-130-192 kubenswrapper[2585]: I0910 21:33:49.022924 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-49tl9_04fa3508-acd2-42fa-8dd4-252b7c001ff9/whereabouts-cni/0.log"